vigiles 2.1.1 → 2.3.0

package/README.md

@@ -8,6 +8,11 @@
   <em>Quis custodiet ipsos custodes?</em> — Who watches the watchmen?
 </p>
 
+<p align="center">
+  <strong>Test &amp; verify your Claude Code harness.</strong><br />
+  vigiles <strong>verifies the references</strong> your instruction files make — linter rules, file paths, scripts, code symbols — and <strong>evals</strong> whether your hooks, skills, and CLAUDE.md actually change what the agent does.
+</p>
+
 <p align="center">
   <a href="https://www.npmjs.com/package/vigiles"><img src="https://img.shields.io/npm/v/vigiles?color=orange" alt="npm version" /></a>
   <a href="https://github.com/zernie/vigiles/actions"><img src="https://img.shields.io/github/actions/workflow/status/zernie/vigiles/ci.yml?branch=main" alt="CI" /></a>
@@ -40,7 +45,7 @@ Reads fine. Four things are wrong:
 3. `npm run typecheck` — script removed from package.json
 4. Service/test pairing — no automated check, just a hope
 
-The agent reads this, trusts it, and writes code based on stale claims nobody verified. vigiles **verifies the references in your instruction files** — that each linter rule exists and is enabled, that every file path and script is real — and meets you at whatever commitment level you want.
+The agent reads this, trusts it, and writes code based on stale claims nobody verified. vigiles **verifies the references in your instruction files** — that each linter rule exists and is enabled, that every file path and script is real, and that referenced **code symbols** (functions, classes, constants) actually exist in the files that define them — and meets you at whatever commitment level you want.
 
 Three levels. Each is independently useful; adopt as far up as you like.
 
@@ -210,19 +215,20 @@ Same monotonicity guarantees as `enforce()` — guards can't be silently removed
 
 ## Verified References
 
-`file()`, `cmd()`, and `ref()` catch stale references at compile time:
+`file()`, `cmd()`, `symbol()`, and `ref()` catch stale references at compile time:
 
 ```typescript
-import { claude, file, cmd, ref, instructions } from "vigiles/spec";
+import { claude, file, cmd, symbol, ref, instructions } from "vigiles/spec";
 
 export default claude({
   sections: {
     architecture: instructions`
       Core engine in ${file("src/compile.ts")}.
+      Compile specs with ${symbol("src/compile.ts", "compileClaude")}.
       Run ${cmd("npm test")} to verify.
       See ${ref("skills/strengthen/SKILL.md")} for the strengthen skill.
     `,
-    // If any path is stale → compile error
+    // If any path / script / symbol is stale → compile error
   },
   // ...
 });
@@ -230,6 +236,12 @@ export default claude({
 
 Skill specs use the same helpers for verified references inside instructions. [Full spec format →](docs/spec-format.md)
 
+### Symbol references (cross-language)
+
+`symbol("file", "name")` (and the markdown mark `` `vigiles:symbol file#name` ``) verify that the named file actually **defines** the symbol — a function, class, method, or constant — parsed with [ast-grep](https://ast-grep.github.io) across **JS/TS, Python, Ruby, Rust, and CSS**. Rename the function and `audit` fails; no project-wide index, no autoloader guessing — it parses the one named file.
+
+In markdown mode the `refs-hook` (PostToolUse) **forces the mark**: it blocks an edit that leaves a code reference bare, telling the agent to write `` `vigiles:symbol path#name` `` or opt out with `<!-- vigiles:ignore -->`. The harness makes the agent mark its references at write time, with full context; `audit` re-verifies them. [Symbol verification →](research/symbol-verification.md)
+
 ## Type-Safe Rule References
 
 `vigiles generate-types` scans your linter configs and emits `.vigiles/generated.d.ts`. With this file, `enforce("eslint/no-consolee")` is a red squiggle in your editor — a typo caught at authoring time, not a runtime surprise. Without it, everything falls back to broad types and still works.
@@ -249,7 +261,10 @@ For markdown frontmatter (Level 1), `vigiles generate-schema` gives the same aut
 ```bash
 npx vigiles init [--target=X.md]    # Scaffold a spec (runs full setup wizard by default)
 npx vigiles compile [files...]      # Compile .spec.ts → .md
-npx vigiles audit [files...]        # Verify hashes + inline/frontmatter/spec rules + coverage
+npx vigiles audit [files...]        # Verify hashes + inline/frontmatter/spec rules + symbols + coverage
+npx vigiles refs <file.md>          # Check the symbol references in an instruction file
+npx vigiles test [files...]         # Run *.harness.mjs deterministic harness tests (no API key)
+npx vigiles eval [files...]         # Run *.eval.mjs real-model harness evals (--trials=N)
 npx vigiles generate-types          # Emit .d.ts from project state (for spec mode)
 npx vigiles generate-types --check  # Verify .d.ts is up to date
 npx vigiles generate-schema         # Emit JSON Schema for vigiles: frontmatter (Level 1)
@@ -323,6 +338,113 @@ Install with [Vercel Skills](https://github.com/vercel-labs/skills): `npx skills
 | `enforce-rules-format` | Validate all rules have enforcement classification                      |
 | `audit-feedback-loop`  | Score your repo's feedback loop maturity                                |
 
+## Test your Claude Code harness
+
+vigiles also ships a library for **testing the harness itself** — your hooks,
+settings, skills, and instruction files. `Agent = Model + Harness`; this tests
+the harness, at three levels.
+
+**Evals — does my change actually move agent behaviour?** Define a fixture, a set
+of **arms** (a hook on vs off, with/without a CLAUDE.md rule), a task, and a
+metric; `runEval` drives the real `claude` CLI N trials per arm and aggregates.
+
+```typescript
+import { runEval, formatEvalReport } from "vigiles/eval";
+
+const report = await runEval({
+  fixture: { "src/billing.ts": "export function chargeCard() {}" },
+  arms: {
+    vanilla: {},
+    gated: { settings: { hooks: { PostToolUse: [refsHook] } } },
+  },
+  task: "Document chargeCard in SKILL.md, referencing it by name.",
+  measure: (ctx) => ({
+    marked: ctx.sh("grep -c vigiles:symbol SKILL.md") !== "0",
+  }),
+  trials: 6,
+});
+console.log(formatEvalReport(report)); //  vanilla marked=0.00   gated marked=0.50
+```
+
+**Deterministic tests — does my hook fire correctly?** No API key, no cost.
+`runHarnessTest` runs real `claude` against a **scripted mock model**
+(`vigiles/mock-model`), so your real hooks fire but the agent's turns are fixed.
+
+```typescript
+import { runHarnessTest, scriptModel } from "vigiles/harness-test";
+
+const r = await runHarnessTest({
+  settings: {
+    hooks: {
+      Stop: [
+        {
+          hooks: [
+            {
+              type: "command",
+              command: "test -f DONE || { echo 'not done' >&2; exit 2; }",
+            },
+          ],
+        },
+      ],
+    },
+  },
+  model: scriptModel([
+    { text: "I'm done" }, // tries to stop → blocked
+    { tool: "Bash", input: { command: "touch DONE" } },
+    { text: "now done" },
+  ]),
+});
+assert(JSON.parse(r.stdout).num_turns > 1); // the Stop hook forced more work
+```
+
+The deterministic tier is reliable for **SessionStart, Stop, UserPromptSubmit,
+and Bash PreToolUse/PostToolUse** hooks — the governance/policy shapes most real
+plugins use; Edit/Write tool-event hooks are headless-gated, so test those at the
+unit tier or via the eval tier.
+
+**Unit-test a hook — no `claude` at all.** A hook is just a process: `runHook`
+pipes an event JSON to its stdin and reports the block/allow decision —
+milliseconds, and the only tier that reaches **every** event (incl. Edit/Write,
+PreCompact, SessionEnd, which the deterministic mock can't trigger).
+
+```typescript
+import { runHook } from "vigiles/run-hook";
+
+const r = runHook(guardCommand, {
+  hook_event_name: "PreToolUse",
+  tool_name: "Bash",
+  tool_input: { command: "git commit --no-verify" },
+});
+assert(r.blocked); // exit 2, decision:"block", or permissionDecision:"deny"
+```
+
+**Run them as a CI command.** `vigiles test` discovers `*.harness.mjs` files
+(deterministic, no API key) and `vigiles eval` discovers `*.eval.mjs` files
+(real model). Canonical, real-plugin-shaped examples to copy:
+
+- [`examples/harness/policy-gate.harness.mjs`](examples/harness/policy-gate.harness.mjs) — a `PreToolUse` Bash policy gate (block `git commit --no-verify`) and a `SessionStart` setup hook, deterministic.
+- [`examples/harness/skill-outcome.eval.mjs`](examples/harness/skill-outcome.eval.mjs) — does a skill change the agent's output? (the question you ask of any `SKILL.md`).
+
+```bash
+npx vigiles test examples/harness/policy-gate.harness.mjs
+npx vigiles eval --trials=6 examples/harness/skill-outcome.eval.mjs
+```
+
+**Test the whole machine.** Point `plugin` at a plugin (or `"./"` for your repo)
+and the real harness — hooks (with `${CLAUDE_PLUGIN_ROOT}` resolved), CLAUDE.md,
+skills, subagents and commands — is loaded into the sandbox, so you test what
+ships, not a retyped subset. `loadPlugin(...).warnings` flags surfaces only a
+real model can drive (subagents, slash commands, MCP), so loading a whole plugin
+never silently tests an empty machine. The library is plain async functions, so
+it runs in **node:test, vitest, or jest** unchanged (shared `expect.extend`
+matchers for the latter two).
+
+[Full guide → `docs/harness-testing.md`](docs/harness-testing.md). The design
+rationale and a coverage assessment against real plugins (protect-mcp,
+obra/superpowers, block-no-verify, 156 wshobson skills) are in
+[`research/harness-testing.md`](research/harness-testing.md); findings from
+running this harness in anger live in [`research/benchmarks-runtime-gates.md`](research/benchmarks-runtime-gates.md).
+
 ## Maturity Levels
 
 From [Feedback Loop Is All You Need](https://zernie.com/blog/feedback-loop-is-all-you-need):

package/dist/action-gate.d.ts

@@ -0,0 +1,28 @@
+import { type RuntimeGate } from "./skill-runtime.js";
+export interface ActionGate {
+    /** Tool name to gate, e.g. "Write" | "Edit" | "Bash". */
+    readonly on: string;
+    /** Deterministic check; a `cmd` command may include `{file}`. */
+    readonly gate: RuntimeGate;
+    /** Optional substring the (JSON-serialized) tool input must contain. */
+    readonly when?: string;
+}
+export interface ActionEvent {
+    /** The tool that just ran (PostToolUse `tool_name`). */
+    readonly tool: string;
+    /** The tool input (`tool_input`), e.g. `{ file_path, command }`. */
+    readonly input?: Record<string, unknown>;
+}
+export interface ActionDecision {
+    readonly allow: boolean;
+    readonly message: string;
+}
+/**
+ * Evaluate action gates against a tool event. Runs every gate whose `on`
+ * matches the tool (and whose `when` substring matches the input); the first
+ * failure blocks. Plan-agnostic — order in any runtime workflow is irrelevant.
+ */
+export declare function evaluateAction(event: ActionEvent, gates: readonly ActionGate[], cwd: string): ActionDecision;
+/** Load action gates from `.vigiles/action-gates.json`. */
+export declare function loadActionGates(cwd: string): ActionGate[];
+//# sourceMappingURL=action-gate.d.ts.map

package/dist/action-gate.js

@@ -0,0 +1,73 @@
+"use strict";
+Object.defineProperty(exports, "__esModule", { value: true });
+exports.evaluateAction = evaluateAction;
+exports.loadActionGates = loadActionGates;
+/**
+ * vigiles — Action gates (the dynamic-workflow reframe).
+ *
+ * A skill gate is bound to a *step* (a fixed position in a plan). When the plan
+ * is generated at runtime (dynamic workflows), the step is the wrong unit. An
+ * **action gate** binds a deterministic check to a *tool action type* instead —
+ * "any time a Write happens to a `.ts` file, eslint must pass on it" — so it
+ * fires regardless of where in the runtime plan the action occurs.
+ *
+ * It is the same deterministic gate primitive (reuses `runGate` + the
+ * author-time reference resolution), re-anchored from step → action. Delivered
+ * as a PostToolUse hook (`vigiles action-hook`): exit 2 blocks the action and
+ * feeds the reason back, exit 0 allows it.
+ *
+ * Config: `.vigiles/action-gates.json` → `{ "gates": [ { on, gate, when? } ] }`.
+ * The gate command may contain `{file}`, substituted with the action's path.
+ */
+const node_fs_1 = require("node:fs");
+const node_path_1 = require("node:path");
+const skill_runtime_js_1 = require("./skill-runtime.js");
+/** The file path an action touched, for `{file}` substitution. */
+function fileOf(event) {
+    const i = event.input ?? {};
+    const v = i.file_path ?? i.path;
+    return typeof v === "string" ? v : "";
+}
+/** Substitute `{file}` in a cmd gate with the action's path. */
+function resolveGate(gate, event) {
+    if (gate.kind !== "cmd" || !gate.command.includes("{file}"))
+        return gate;
+    return { ...gate, command: gate.command.replaceAll("{file}", fileOf(event)) };
+}
+/**
+ * Evaluate action gates against a tool event. Runs every gate whose `on`
+ * matches the tool (and whose `when` substring matches the input); the first
+ * failure blocks. Plan-agnostic — order in any runtime workflow is irrelevant.
+ */
+function evaluateAction(event, gates, cwd) {
+    const inputStr = JSON.stringify(event.input ?? "");
+    for (const g of gates) {
+        if (g.on !== event.tool)
+            continue;
+        if (g.when && !inputStr.includes(g.when))
+            continue;
+        const outcome = (0, skill_runtime_js_1.runGate)(resolveGate(g.gate, event), cwd);
+        if (!outcome.ok) {
+            const tail = outcome.output ? `\n${outcome.output}` : "";
+            return {
+                allow: false,
+                message: `Action gate failed after ${event.tool}: ${(0, skill_runtime_js_1.gateLabel)(g.gate)} did not pass.${tail}`,
+            };
+        }
+    }
+    return { allow: true, message: "" };
+}
+/** Load action gates from `.vigiles/action-gates.json`. */
+function loadActionGates(cwd) {
+    const p = (0, node_path_1.resolve)(cwd, ".vigiles/action-gates.json");
+    if (!(0, node_fs_1.existsSync)(p))
+        return [];
+    try {
+        const parsed = JSON.parse((0, node_fs_1.readFileSync)(p, "utf-8"));
+        return Array.isArray(parsed.gates) ? parsed.gates : [];
+    }
+    catch {
+        return [];
+    }
+}
+//# sourceMappingURL=action-gate.js.map