vigiles 2.6.0 → 3.0.0

package/skills/test-harness/SKILL.md

@@ -0,0 +1,157 @@
+---
+name: test-harness
+description: Install vigiles and test a Claude Code harness — hooks, skills, settings, CLAUDE.md — by picking the right tier (unit / deterministic / eval) and writing a test that passes. Use when the user wants to check that a hook fires or blocks, that a skill triggers, that injected context lands, or that a harness change moves what the agent does.
+---
+
+Test the Claude Code **harness** — the hooks, skills, settings, and CLAUDE.md that
+steer an agent — as the assembled machine it ships as. vigiles gives three tiers,
+cheapest first; this skill picks the right one, writes the test, and runs it.
+
+The guiding rule: **start at the cheapest tier that can answer the question, and
+climb only when it genuinely can't.** Two of the three tiers need no model and no
+API key, so they run on every commit for free — reach for the paid real-model
+tier only when the question actually requires a real model.
+
+## Step 0 — Pick the tier (the judgment call)
+
+Match what you're testing to the cheapest tier that can answer it:
+
+| What you're testing                                                                                                                    | Tier              | Cost                                             | API                                                                                           |
+| -------------------------------------------------------------------------------------------------------------------------------------- | ----------------- | ------------------------------------------------ | --------------------------------------------------------------------------------------------- |
+| "Does this hook block/allow event X?" — pure hook logic, **every** event type (incl. Edit/Write, PreCompact, SessionEnd, SubagentStop) | **Unit**          | free, milliseconds, no `claude`                  | `runHook`                                                                                     |
+| "Is the hook actually **wired into** the assembled plugin and does it fire in a real session?"                                         | **Deterministic** | free, no API key (real `claude` + scripted mock) | `runHarnessTest` + `scriptModel`                                                              |
+| "Did the injected context (a SessionStart hook, a `/command`) actually **reach the model**?"                                           | **Deterministic** | free, no API key                                 | `runHarnessTest` → `trace.modelRequests` / `assertRequestContains`                            |
+| "Does this skill's **description trigger** when it should (recall) **and stay quiet** when it shouldn't (precision)?"                  | **Eval**          | **paid** (real model)                            | `measureTriggerRate` (+ `irrelevantPrompts`) → `assertTriggerRate({ min, maxFalsePositive })` |
+| "Does this harness change **move what the agent does**?" (A/B, signal vs noise)                                                        | **Eval**          | **paid** (real model)                            | `runEval` + `assertSignificant`                                                               |
+
+Most harness questions — block/allow, wired-in, context-landed — never need a
+model. Only "does the model trigger / behave differently" needs the eval tier.
+
+If the unit and deterministic tiers can both answer it, **prefer unit**: it's
+faster and reaches events the deterministic mock can't drive.
+
+## Step 1 — Ensure vigiles is installed
+
+Check whether `vigiles` is a dependency (`package.json`), and install it as a
+dev dependency if not:
+
+```bash
+npm i -D vigiles    # or: pnpm add -D vigiles / yarn add -D vigiles
+```
+
+The deterministic tier additionally needs the `claude` CLI on PATH (no API key):
+`npm i -g @anthropic-ai/claude-code`. The eval tier needs model auth. If the
+`claude` CLI is missing, you can still write and run **unit**-tier tests.
+
+## Step 2 — Locate the harness surface to test
+
+Find what the project actually ships, in this order:
+
+1. `.claude/settings.json` / `.claude/settings.local.json` — inline `hooks`.
+2. `.claude-plugin/plugin.json` — a plugin manifest (`hooks`, `skills`, `agents`, `mcpServers`).
+3. `hooks/hooks.json` — the plugin hooks convention (e.g. obra/superpowers).
+4. `skills/<name>/SKILL.md`, `agents/<name>.md`, `commands/<name>.md`.
+
+Pick one concrete thing to pin down — a specific `PreToolUse` hook, a specific
+`SessionStart` injection, a specific skill.
+
+## Step 3 — Write the test for the chosen tier
+
+**Unit (`runHook`)** — hand a hook a synthesized event, assert the decision:
+
+```ts
+import { runHook, assertHookBlocked } from "vigiles/testing";
+
+const r = runHook(hookCommand, {
+  hook_event_name: "PreToolUse",
+  tool_name: "Bash",
+  tool_input: { command: "git commit --no-verify" },
+});
+assertHookBlocked(r); // exit 2 / decision:"block" / permissionDecision:"deny"
+```
+
+Testing a hook you didn't write (a vendored third-party script)? Mark it
+`{ trusted: false }` and it runs confined under bubblewrap by default (read-only
+host, cleared env, no network egress). Add `{ recordEgress: true }` to also
+**record** what it tries to reach — `r.egress` plus `assertNoEgress(r)` /
+`assertEgressOnly(r, [...])` — the supply-chain check for "what does this skill
+phone home to / install from?". When the hook's setup needs a _real_ install,
+`{ egress: { allow: ["registry.npmjs.org"] } }` lets it reach only that
+allowlist (a packet-layer `nft` wall, so a raw socket off-list is dropped too) →
+`r.egress` (allowed hosts) + `r.egressDropped`. Be precise about the boundaries:
+see
+[`docs/sandboxing.md`](../../docs/sandboxing.md) (it blocks destruction and
+egress, but does NOT isolate reads of host files, and only under bwrap).
+
+**Deterministic (`runHarnessTest`)** — load the real plugin, drive a scripted
+mock model, assert the hook fired (or the context landed):
+
+```ts
+import {
+  runHarnessTest,
+  scriptModel,
+  assertHookFired,
+  assertRequestContains,
+} from "vigiles/testing";
+
+const r = await runHarnessTest({
+  pluginDir: "./", // or { settings: { hooks: {...} } }
+  transcript: true,
+  model: scriptModel([{ text: "ok" }]),
+});
+assertHookFired(r, "SessionStart");
+assertRequestContains(r, "expected injected text"); // did it actually land?
+```
+
+**Eval (`runEval`)** — A/B the change on vs off across real-model trials, then
+gate on significance, not eyeballing:
+
+```ts
+import { runEval, assertSignificant } from "vigiles/testing";
+
+const report = await runEval({
+  arms: { off: {}, on: { pluginDir: "./" } },
+  task: "…a task the harness change should affect…",
+  measure: (ctx) => ({ ok: /* a bare predicate over the trace */ true }),
+  trials: 6,
+  cache: "readwrite",
+});
+assertSignificant(report, { baseline: "off", arm: "on", metric: "ok" });
+```
+
+## Step 4 — Run it
+
+In a runner (node:test / vitest / jest) the tests are plain async functions. Or
+use the zero-setup CLI, which discovers and runs the files:
+
+```bash
+npx vigiles test                 # *.harness.{mjs,ts} — unit + deterministic, no API key
+npx vigiles eval --trials=6      # *.eval.{mjs,ts} — real model (local / nightly, not CI)
+```
+
+Unit-tier `runHook` tests need no `claude` and **always run** — write and run them
+even with no `claude` installed. A tier that genuinely can't run reports a loud
+`⊘ SKIPPED` (tallied separately, never a fake `✓`); a standalone script emits one
+via `skip(reason)` from `vigiles/testing`. A skip passes by default, but in a CI
+job that asserts the capability is present, run **`vigiles test --no-skip`** so a
+skipped tier fails — a green-with-skips is untested surface. Keep unit +
+deterministic tests in CI (free); run evals locally or on a schedule with auth.
+
+## When the user didn't say what to test
+
+Don't ask them to specify — **pick something real and demonstrate.** Scan the
+harness surface (Step 2), choose the cheapest meaningful test, write it, run it,
+and show the result. Good default picks, in order:
+
+1. A `PreToolUse` hook → **unit-test** that it blocks the thing it's meant to block (and allows a safe sibling).
+2. A `SessionStart` hook that injects context → **deterministic** test that the text actually reaches the model (`assertRequestContains`).
+3. A skill → **deterministic** test that it resolves via `pluginDir`, then offer the paid `measureTriggerRate` eval as a follow-up.
+
+Then say which tier you used and why, and offer to climb a tier if the cheaper
+test can't fully answer their question.
+
+## Reference
+
+The full guide — every tier, testing skills for real, "fired ≠ landed", the
+safe-by-default sandbox, the coverage matrix, and how it compares to promptfoo —
+is in [`docs/harness-testing.md`](../../docs/harness-testing.md).

package/dist/action.d.ts

@@ -1,7 +0,0 @@
-/**
- * GitHub Action entry point for vigiles.
- *
- * Runs `compile` or `audit` depending on the action input.
- */
-export {};
-//# sourceMappingURL=action.d.ts.map

package/dist/action.js

@@ -1,180 +0,0 @@
-"use strict";
-/**
- * GitHub Action entry point for vigiles.
- *
- * Runs `compile` or `audit` depending on the action input.
- */
-Object.defineProperty(exports, "__esModule", { value: true });
-const node_fs_1 = require("node:fs");
-const node_path_1 = require("node:path");
-const node_child_process_1 = require("node:child_process");
-const glob_1 = require("glob");
-const compile_js_1 = require("./compile.js");
-// ---------------------------------------------------------------------------
-// Read action inputs
-// ---------------------------------------------------------------------------
-const command = process.env["INPUT_COMMAND"] ?? "audit";
-const pathsInput = process.env["INPUT_PATHS"];
-const maxRulesInput = process.env["INPUT_MAX-RULES"] ?? process.env["INPUT_MAX_RULES"];
-const catalogOnly = (process.env["INPUT_CATALOG-ONLY"] ??
-    process.env["INPUT_CATALOG_ONLY"] ??
-    "false") === "true";
-const maxRules = maxRulesInput ? Number(maxRulesInput) : undefined;
-// ---------------------------------------------------------------------------
-// Spec loading (from compiled dist/)
-// ---------------------------------------------------------------------------
-async function loadSpec(specPath) {
-    // Try several resolution strategies:
-    // 1. Replace /src/ with /dist/ (standard TS project layout)
-    // 2. Look under dist/ directly (root-level specs like CLAUDE.md.spec.ts)
-    // 3. Use the .js extension in-place (pre-compiled specs)
-    const jsName = specPath.replace(/\.ts$/, ".js");
-    const candidates = [
-        (0, node_path_1.resolve)(process.cwd(), jsName.replace(/\/src\//, "/dist/")),
-        (0, node_path_1.resolve)(process.cwd(), "dist", jsName),
-        (0, node_path_1.resolve)(process.cwd(), jsName),
-    ];
-    // Deduplicate (candidate 1 and 3 may match when there's no /src/)
-    const unique = [...new Set(candidates)];
-    for (const distPath of unique) {
-        if (!(0, node_fs_1.existsSync)(distPath))
-            continue;
-        try {
-            const mod = (await import(distPath));
-            // CJS modules (TypeScript with module: "Node16", no "type":
-            // "module") produce `exports["default"] = spec`, so dynamic
-            // import wraps it as `{ default: { default: spec } }`. Unwrap
-            // the double-default when present.
-            const raw = mod.default;
-            if (raw && typeof raw === "object" && "default" in raw) {
-                return raw.default;
-            }
-            return raw;
-        }
-        catch {
-            continue;
-        }
-    }
-    return null;
-}
-// ---------------------------------------------------------------------------
-// Commands
-// ---------------------------------------------------------------------------
-async function runCompile() {
-    const specs = pathsInput
-        ? pathsInput
-            .split(",")
-            .map((p) => p.trim())
-            .filter(Boolean)
-        : (0, glob_1.globSync)("**/*.spec.ts", { ignore: ["node_modules/**", "dist/**"] });
-    if (specs.length === 0) {
-        console.log("No .spec.ts files found.");
-        return true;
-    }
-    let allValid = true;
-    const basePath = process.cwd();
-    for (const specPath of specs) {
-        const spec = await loadSpec(specPath);
-        if (!spec) {
-            console.log(`::error file=${specPath}::Failed to load spec`);
-            allValid = false;
-            continue;
-        }
-        if (spec._specType === "claude") {
-            const primaryOutput = specPath.replace(/\.spec\.ts$/, "");
-            const { markdown, errors, targets } = (0, compile_js_1.compileClaude)(spec, {
-                basePath,
-                specFile: specPath,
-                maxRules,
-                catalogOnly,
-            });
-            if (errors.length > 0) {
-                for (const err of errors) {
-                    console.log(`::error file=${specPath}::${err.message}`);
-                }
-                allValid = false;
-            }
-            // Write primary target
-            (0, node_fs_1.writeFileSync)((0, node_path_1.resolve)(basePath, primaryOutput), markdown);
-            const outputNames = [primaryOutput];
-            // Write additional targets with swapped heading + recomputed hash
-            for (const t of targets.slice(1)) {
-                const body = markdown
-                    .replace(/^<!-- vigiles:[^\n]+\n\n?/, "")
-                    .replace(/^# [^\n]+/, `# ${t}`);
-                const additional = (0, compile_js_1.addHash)(body, specPath);
-                const dir = primaryOutput.substring(0, primaryOutput.lastIndexOf("/") + 1);
-                const targetPath = dir + t;
-                (0, node_fs_1.writeFileSync)((0, node_path_1.resolve)(basePath, targetPath), additional);
-                outputNames.push(targetPath);
-            }
-            console.log(`Compiled: ${specPath} → ${outputNames.join(", ")}`);
-        }
-        else if (spec._specType === "skill") {
-            const outputPath = specPath.replace(/\.spec\.ts$/, "");
-            const { markdown, errors } = (0, compile_js_1.compileSkill)(spec, {
-                basePath,
-                specFile: specPath,
-            });
-            if (errors.length > 0) {
-                for (const err of errors) {
-                    console.log(`::error file=${specPath}::${err.message}`);
-                }
-                allValid = false;
-            }
-            (0, node_fs_1.writeFileSync)((0, node_path_1.resolve)(basePath, outputPath), markdown);
-            console.log(`Compiled: ${specPath} → ${outputPath}`);
-        }
-    }
-    return allValid;
-}
-function runAudit() {
-    // Delegate to the CLI so the action always runs the full audit flow
-    // (hash verification, spec validation, duplicate detection, coverage
-    // report, strengthen suggestions). Previously this only ran
-    // checkFileHash, which silently skipped everything else.
-    const files = pathsInput
-        ? pathsInput
-            .split(",")
-            .map((p) => p.trim())
-            .filter(Boolean)
-        : [];
-    const cliPath = (0, node_path_1.resolve)(__dirname, "cli.js");
-    const args = ["audit", ...files];
-    const result = (0, node_child_process_1.spawnSync)(process.execPath, [cliPath, ...args], {
-        stdio: "inherit",
-        env: {
-            ...process.env,
-            // Ensure the CLI emits GitHub annotations during the action run
-            GITHUB_ACTIONS: process.env.GITHUB_ACTIONS ?? "true",
-        },
-    });
-    if (result.error) {
-        console.log(`::error::Failed to run vigiles audit: ${result.error.message}`);
-        return false;
-    }
-    // Exit codes: 0 clean, 1 warnings, 2 hard errors.
-    return result.status === 0;
-}
-// ---------------------------------------------------------------------------
-// Main
-// ---------------------------------------------------------------------------
-void (async () => {
-    let valid;
-    if (command === "compile") {
-        valid = await runCompile();
-    }
-    else if (command === "audit") {
-        valid = runAudit();
-    }
-    else {
-        console.log(`::error::Unknown vigiles command "${command}". Valid commands: compile, audit.`);
-        process.exit(1);
-    }
-    console.log(`::set-output name=valid::${String(valid)}`);
-    if (!valid) {
-        console.log("::error::vigiles failed — see errors above");
-        process.exit(1);
-    }
-})();
-//# sourceMappingURL=action.js.map

package/dist/refs.d.ts

@@ -1,44 +0,0 @@
-/** An inline code span with its 1-based source line. */
-export interface Span {
-    readonly text: string;
-    readonly line: number;
-}
-/**
- * Extract inline code spans, skipping fenced code blocks (R1). Returns each
- * span's trimmed text and 1-based line.
- */
-export declare function inlineSpans(markdown: string): Span[];
-/** A parsed file-qualified reference. */
-export interface SymbolRef {
-    readonly file: string;
-    readonly symbol: string;
-    readonly line: number;
-}
-/** A reference that failed verification. */
-export interface SymbolRefError extends SymbolRef {
-    readonly reason: string;
-}
-/** Extract the `vigiles:symbol` references from a markdown file. */
-export declare function symbolRefs(markdown: string): SymbolRef[];
-/**
- * Verify the file-qualified symbol references in a markdown file: the named
- * file must exist and define the named symbol. `basePath` is the directory the
- * paths resolve against (the instruction file's own directory).
- */
-export declare function verifySymbolRefs(markdown: string, basePath: string): SymbolRefError[];
-/**
- * Whether a span looks like a *code reference* that ought to carry a
- * file-qualified mark — a scoped name, or an identifier that isn't a bare
- * lowercase prose word. A function-call form `` `foo(args)` `` is treated as a
- * reference to its callee `foo`. Paths/filenames are excluded (they are `file`
- * refs).
- */
-export declare function isCodeShaped(text: string): boolean;
-/**
- * Code-shaped inline references that are NOT yet marked — the spans the
- * enforcement hook makes the agent mark as `` `vigiles:symbol path.ext#symbol` ``
- * or opt out of with `<!-- vigiles:ignore -->` (or `<!-- vigiles:ignore-file -->`
- * for the whole file).
- */
-export declare function unmarkedCodeRefs(markdown: string): Span[];
-//# sourceMappingURL=refs.d.ts.map

package/dist/run-scripts.d.ts

@@ -1,20 +0,0 @@
-export interface ScriptRunResult {
-    readonly file: string;
-    readonly code: number;
-}
-/**
- * Expand the given path/glob patterns into concrete script files. A pattern
- * that is an existing file passes through unchanged; anything else is treated
- * as a glob. Falls back to `defaultGlob` when no patterns are given. Results
- * are deduped and sorted; `node_modules` and `dist` are always ignored.
- */
-export declare function discoverScripts(patterns: readonly string[], defaultGlob: string, cwd: string): string[];
-/**
- * Run each script as `node <file>`, inheriting stdio so the script's own report
- * streams to the console. `env` is merged over `process.env` for every child
- * (e.g. `VIGILES_TRIALS`). Returns the per-file exit codes.
- */
-export declare function runScripts(files: readonly string[], cwd: string, env?: NodeJS.ProcessEnv): ScriptRunResult[];
-/** Format a one-line-per-file run summary with a pass/fail tally. */
-export declare function formatScriptSummary(results: readonly ScriptRunResult[]): string;
-//# sourceMappingURL=run-scripts.d.ts.map

package/dist/run-scripts.js

@@ -1,70 +0,0 @@
-"use strict";
-Object.defineProperty(exports, "__esModule", { value: true });
-exports.discoverScripts = discoverScripts;
-exports.runScripts = runScripts;
-exports.formatScriptSummary = formatScriptSummary;
-/**
- * vigiles — run harness-test / eval script files via the CLI.
- *
- * `vigiles test` and `vigiles eval` discover `*.harness.mjs` / `*.eval.mjs`
- * scripts and run each as a child `node` process, so the two-tier
- * harness-testing API (`src/harness-test.ts`, `src/eval.ts`) works as a CI
- * command, not just `node x.mjs`. The scripts stay plain Node modules (they
- * import from the built `dist/`), so they also run standalone — the CLI just
- * discovers, runs, and aggregates exit codes.
- */
-const node_child_process_1 = require("node:child_process");
-const node_path_1 = require("node:path");
-const node_fs_1 = require("node:fs");
-const glob_1 = require("glob");
-/**
- * Expand the given path/glob patterns into concrete script files. A pattern
- * that is an existing file passes through unchanged; anything else is treated
- * as a glob. Falls back to `defaultGlob` when no patterns are given. Results
- * are deduped and sorted; `node_modules` and `dist` are always ignored.
- */
-function discoverScripts(patterns, defaultGlob, cwd) {
-    const globs = patterns.length > 0 ? patterns : [defaultGlob];
-    const found = new Set();
-    for (const p of globs) {
-        if ((0, node_fs_1.existsSync)((0, node_path_1.resolve)(cwd, p))) {
-            found.add(p);
-            continue;
-        }
-        for (const m of (0, glob_1.globSync)(p, {
-            cwd,
-            ignore: ["node_modules/**", "dist/**"],
-        })) {
-            found.add(m);
-        }
-    }
-    return [...found].sort();
-}
-/**
- * Run each script as `node <file>`, inheriting stdio so the script's own report
- * streams to the console. `env` is merged over `process.env` for every child
- * (e.g. `VIGILES_TRIALS`). Returns the per-file exit codes.
- */
-function runScripts(files, cwd, env = {}) {
-    const results = [];
-    for (const file of files) {
-        const res = (0, node_child_process_1.spawnSync)("node", [file], {
-            cwd,
-            stdio: "inherit",
-            env: { ...process.env, ...env },
-        });
-        results.push({ file, code: res.status ?? 1 });
-    }
-    return results;
-}
-/** Format a one-line-per-file run summary with a pass/fail tally. */
-function formatScriptSummary(results) {
-    const lines = results.map((r) => `  ${r.code === 0 ? "✓" : "✗"} ${r.file}` +
-        (r.code === 0 ? "" : ` (exit ${String(r.code)})`));
-    const failed = results.filter((r) => r.code !== 0).length;
-    lines.push(failed === 0
-        ? `\n${String(results.length)} passed.`
-        : `\n${String(failed)}/${String(results.length)} failed.`);
-    return lines.join("\n");
-}
-//# sourceMappingURL=run-scripts.js.map