vigiles 2.4.0 → 2.6.0

package/README.md

@@ -9,8 +9,8 @@
 </p>
 
 <p align="center">
-  <strong>Test &amp; verify your Claude Code harness.</strong><br />
-  vigiles <strong>verifies the references</strong> your instruction files make — linter rules, file paths, scripts, code symbols — and <strong>evals</strong> whether your hooks, skills, and CLAUDE.md actually change what the agent does.
+  <strong>The missing linting + testing layer for agentic coding.</strong><br />
+  vigiles <strong>lints</strong> the references your instruction files make — linter rules, file paths, scripts, code symbols — and <strong>tests</strong> whether your hooks, skills, and CLAUDE.md actually change what the agent does.
 </p>
 
 <p align="center">
@@ -19,30 +19,39 @@
   <a href="https://github.com/zernie/vigiles/blob/main/LICENSE"><img src="https://img.shields.io/github/license/zernie/vigiles" alt="License" /></a>
 </p>
 
+<p align="center">
+  <b>Two pillars →</b>
+  &nbsp;<a href="#verify-your-instruction-files">① Verify your instruction files</a>
+  &nbsp;·&nbsp;
+  <a href="#test-your-claude-code-harness">② Test your Claude Code harness</a>
+</p>
+
 ---
 
 <details>
 <summary><b>Contents</b></summary>
 
-**Pillar 1 — verify your instruction files** · references your CLAUDE.md makes that a linter, the filesystem, and package.json can prove
+- [**Two pillars — pick one or both**](#two-pillars--pick-one-or-both)
+- **Pillar 1** — [verify your instruction files](#verify-your-instruction-files) · full guide: [docs/verifying-instruction-files.md](docs/verifying-instruction-files.md)
+- **Pillar 2** — [test your Claude Code harness](#test-your-claude-code-harness) · full guide: [docs/harness-testing.md](docs/harness-testing.md)
+- [Quick start](#quick-start) · [CLI & CI](#cli--ci) · [Skills](#skills) · [Related tools](#related-tools)
 
-- Three adoption levels: [inline comments](#level-0--inline-comments-30-seconds-no-new-files) → [YAML frontmatter](#level-1--yaml-frontmatter-editor-autocomplete-still-no-typescript) → [typed spec](#level-2--typed-spec-compiler-grade-guarantees)
-- [What changes with vigiles](#what-changes-with-vigiles)
-- [Quick start](#quick-start)
-- [Three rule types](#three-rule-types) — `enforce` / `guidance` / `guard`
-- [Verified references](#verified-references) — `file` / `cmd` / `symbol` / `ref`
+</details>
 
-**Pillar 2 — [test your Claude Code harness](#test-your-claude-code-harness)** · eval whether your hooks, skills, and CLAUDE.md actually change what the agent does
+## Two pillars — pick one or both
 
-- [Level 1 — unit-test a hook (no AI)](#level-1--test-a-hook-by-itself-no-ai-milliseconds)
-- [Level 2 — does it fire in a real session?](#level-2--does-it-fire-in-a-real-session-free-scripted-ai)
-- [Level 3 — does it change behaviour?](#level-3--does-it-change-what-claude-does-real-ai-occasional)
-- [Test skills for real + assert on actions](#test-your-skills-for-real--and-assert-on-what-claude-did)
-- [Run them in CI](#run-them-in-ci)
+An agent runs real commands in your repo — it can delete the wrong files, leak a secret, or burn tokens looping on a stale instruction nobody checked. You'd never ship an app without a linter and a test suite. An AI agent steering your codebase is no different — so why is its harness trusted on vibes?
 
-**More** — [CLI & CI](#cli--ci) · [Skills](#skills) · [Maturity levels](#maturity-levels) · [Related tools](#related-tools)
+`Agent = Model + Harness`. Your harness is everything that steers a run — the **instructions** you write _and_ the **hooks, skills, and settings** that enforce them. vigiles is the **missing linting + testing layer for agentic coding**: it **lints** your instruction files and **tests** your harness. Two pillars of equal weight — adopt either on its own, or both:
 
-</details>
+|       | Pillar                                                              | What it does                                                                                                                                                                 |
+| ----- | ------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| **①** | [**Verify your instruction files**](#verify-your-instruction-files) | Every linter rule, file path, script, and code symbol your CLAUDE.md cites is checked against reality, so stale references can't silently mislead the agent.                 |
+| **②** | [**Test your harness**](#test-your-claude-code-harness)             | Your hooks and skills are code — vigiles tests that they actually fire, **deterministically and for free** (no model, no API key) before you ever pay for a real-model eval. |
+
+They share the thesis but not a dependency: verify your instructions without ever writing a harness test, or test your harness without a single `.spec.ts`. Pick the pillar that hurts today.
+
+## Verify your instruction files
 
 Your CLAUDE.md lies to your agent. Here's the fix.
 
@@ -70,13 +79,11 @@ Reads fine. Four things are wrong:
 
 The agent reads this, trusts it, and writes code based on stale claims nobody verified. vigiles **verifies the references in your instruction files** — that each linter rule exists and is enabled, that every file path and script is real, and that referenced **code symbols** (functions, classes, constants) actually exist in the files that define them — and meets you at whatever commitment level you want.
 
-> **See it in 60 seconds:** `npm run demo` runs `vigiles audit` against a deliberately-broken instruction file and catches a renamed symbol and a missing MCP tool (_"did you mean `purge`?"_), while the truthful references pass silently. [examples/demo →](examples/demo)
+> **Set it up — paste this into Claude Code:**
+>
+> > Install vigiles and verify my instruction files. Scan my CLAUDE.md / AGENTS.md, turn its real claims into checked references (linter rules, file paths, scripts, symbols), run `vigiles audit`, and show me what's already stale. Use sensible defaults, but **ask me first** whether to stay in markdown mode or generate a typed `.spec.ts`, and whether to wire the audit into CI and install the edit-blocking hooks.
 
-Three levels. Each is independently useful; adopt as far up as you like.
-
-### Level 0 — inline comments (30 seconds, no new files)
-
-Add a comment to your existing CLAUDE.md and audit it:
+Or do it by hand — add a marker to your existing CLAUDE.md and audit it, no install, no new files:
 
 ```md
 <!-- vigiles:enforce eslint/no-console "Route output through logger.ts" -->
@@ -86,181 +93,46 @@ Add a comment to your existing CLAUDE.md and audit it:
 npx vigiles audit CLAUDE.md
 ```
 
-Each rule is checked against your real linter config — typos get closest-match suggestions, disabled rules are flagged. Zero install commitment, zero new files.
-
-### Level 1 — YAML frontmatter (editor autocomplete, still no TypeScript)
-
-Promote your rules into a `vigiles:` block at the top of the file:
-
-```yaml
----
-# yaml-language-server: $schema=./.vigiles/schema.json
-vigiles:
-  enforce:
-    - rule: "@typescript-eslint/no-explicit-any"
-      why: "Use unknown and narrow with type guards."
-    - rule: eslint/no-console
-      why: "Route output through logger.ts"
----
-```
-
-`npx vigiles generate-schema` emits a JSON Schema from your project's _actual_ enabled rules, so your editor's built-in YAML language server (VS Code, JetBrains, neovim) autocompletes rule names and red-squiggles typos — at edit time, with no TypeScript in the project. `vigiles audit` enforces the same rules in CI. [Markdown mode →](docs/markdown-mode.md)
-
-### Level 2 — typed spec (compiler-grade guarantees)
-
-When you want the strongest guarantees, compile a typed spec. Every linter rule reference is verified against your real config, every file path against the filesystem, every npm script against package.json. Stale references become compile errors — caught at edit time, not when the agent silently ignores you.
-
-```typescript
-// CLAUDE.md.spec.ts
-import { claude, enforce, guidance } from "vigiles/spec";
-
-export default claude({
-  commands: {
-    "npm run build": "Compile TypeScript to dist/",
-    "npm test": "Build and run all tests",
-    // ✗ "npm run typecheck" → compile error: script not in package.json
-  },
-
-  keyFiles: {
-    "src/utils/narrowing.ts": "Type guard utilities",
-    // ✗ "src/utils/type-helpers.ts" → compile error: file not found
-  },
-
-  rules: {
-    "no-explicit-any": enforce(
-      "@typescript-eslint/no-explicit-any",
-      "Use unknown and narrow with type guards.",
-    ),
-    // ✗ if rule is disabled in config → compile error
-
-    "research-first": guidance("Google unfamiliar APIs first."),
-  },
-});
-```
-
-```bash
-$ npx vigiles compile
-
-✓ CLAUDE.md.spec.ts → CLAUDE.md
-  2 rules (1 linter-verified, 1 guidance)
-  ~180 tokens
-```
-
-At this level the spec is the source of truth and CLAUDE.md is a build artifact. The agent edits the spec — hooks auto-compile, types catch typos in the editor, CI catches drift.
-
-Companion repo for [Feedback Loop Is All You Need](https://zernie.com/blog/feedback-loop-is-all-you-need).
+Each reference is checked against reality — a typo gets a closest-match suggestion, a disabled rule is flagged. That's **markdown mode**; step up to a **typed spec** (`.spec.ts` → compiled CLAUDE.md, compiler-grade guarantees) when you want it.
 
-## What Changes With vigiles
-
-### Claude Code
-
-|                                     | Without vigiles              | With vigiles                                                   |
-| ----------------------------------- | ---------------------------- | -------------------------------------------------------------- |
-| **Instructions**                    | Hand-written CLAUDE.md       | Compiled from `.spec.ts` (build artifact)                      |
-| **Linter rule references**          | Trust-based (nobody checks)  | Verified at compile time against real config                   |
-| **File paths**                      | Rot silently when renamed    | `file()` references checked against filesystem                 |
-| **Commands**                        | Stale scripts go unnoticed   | `cmd()` references checked against package.json                |
-| **Direct edits to CLAUDE.md**       | Anyone can, nobody knows     | PreToolUse hook blocks edits, redirects to spec                |
-| **Spec edits**                      | N/A                          | PostToolUse hook auto-compiles to markdown                     |
-| **Linter config changes**           | CLAUDE.md drifts out of sync | PostToolUse hook auto-regenerates types                        |
-| **guidance → enforce upgrades**     | Manual guesswork             | `/strengthen` reads per-linter docs, suggests upgrades         |
-| **New lint rules from PR feedback** | Copy-paste from review       | `/pr-to-lint-rule` generates rule + tests + spec entry         |
-| **CI**                              | Nothing to verify            | `vigiles audit` catches hand-edits, disabled rules, stale refs |
-
-<details>
-<summary><b>Codex</b> (same compile-time checks, no hooks)</summary>
-
-|                               | Without vigiles                  | With vigiles                                            |
-| ----------------------------- | -------------------------------- | ------------------------------------------------------- |
-| **Instructions**              | Hand-written AGENTS.md           | Compiled from `.spec.ts`                                |
-| **Linter rule references**    | Trust-based                      | Verified at compile time                                |
-| **File paths / commands**     | Rot silently                     | Checked at compile time                                 |
-| **Direct edits to AGENTS.md** | Undetected                       | CI catches hash mismatch                                |
-| **Hooks / auto-compile**      | Not available (no plugin system) | Not available — run `vigiles compile` manually or in CI |
-| **CI**                        | Nothing to verify                | Same `vigiles audit` pipeline as Claude                 |
-
-</details>
-
-Everything vigiles compiles and audits is **deterministic** — same input, same output, no LLM in the loop. The non-deterministic parts (authoring specs, suggesting upgrades, writing custom rules) are agent skills that run outside the compilation pipeline. [Determinism breakdown and flow diagram →](docs/comparison.md)
+→ **Full guide: [docs/verifying-instruction-files.md](docs/verifying-instruction-files.md)** — the adoption ladder, the three rule types (`enforce` / `guidance` / `guard`), verified references (`file` / `cmd` / `symbol` / `ref`), and the before/after tables.
 
 ## Quick Start
 
-The fastest path is markdown mode — add a marker to your existing CLAUDE.md and audit it, no install or new files (see [Level 0 / Level 1](#level-0--inline-comments-30-seconds-no-new-files) above and [docs/markdown-mode.md](docs/markdown-mode.md)). When you want compiler-grade guarantees, scaffold a typed spec:
-
 ```bash
 npx vigiles init
 ```
 
-The wizard auto-detects your project, creates a spec, scans your linters, compiles to markdown, adds a CI step, and installs Claude Code hooks. After install: the agent edits the spec (hooks block direct CLAUDE.md edits), the spec auto-compiles on save, and `vigiles audit` catches drift in CI.
-
-Start with `guidance()` rules (zero config). When you're ready, run `/strengthen` to find rules that can be upgraded to compile-verified `enforce()`. Already have a hand-written CLAUDE.md? The wizard detects it and offers migration. Flags (`--strict`, `--target=AGENTS.md`, `--no-gha`) and non-interactive agent usage are in the [CLI reference](docs/cli.md) and [agent setup guide](docs/agent-setup.md).
-
-## Three Rule Types
-
-**`enforce()`** — delegated to a linter. vigiles verifies the rule exists in the catalog AND is enabled in your project config. A disabled rule is a compile error.
-
-<!-- vigiles:ignore -->
-
-```typescript
-"no-any":    enforce("@typescript-eslint/no-explicit-any", "Use unknown and narrow."),
-"no-print":  enforce("ruff/T201", "Use logging module."),
-"no-unwrap": enforce("clippy/unwrap_used", "Use expect() with context."),
-```
-
-Supports ESLint, Stylelint, Ruff, Clippy, Pylint, RuboCop, and Cedar policies. [Full linter support details →](docs/linter-support.md)
-
-**`guidance()`** — prose advice. No mechanical enforcement, but not untracked: guidance rules participate in the monotonicity proof system. Once a rule exists, it can be strengthened ( `guidance` → `enforce` ) but never weakened or removed without an explicit allowlist. This prevents silent erosion of conventions over time.
-
-```typescript
-"research-first": guidance("Google unfamiliar APIs first."),
-```
-
-**`guard()`** — reactive: runs a command when watched files change (e.g. `*.spec.ts` → `npx vigiles compile`). One declaration emits hooks for every supported system (Claude Code PostToolUse, husky pre-commit, etc.) — no copy-pasting the same trigger across `.claude/settings.json`, `.husky/`, and CI. Same monotonicity guarantees as `enforce()`. [Full spec format →](docs/spec-format.md)
-
-## Verified References
-
-`file()`, `cmd()`, `symbol()`, and `ref()` catch stale references at compile time:
-
-```typescript
-import { claude, file, cmd, symbol, ref, instructions } from "vigiles/spec";
-
-export default claude({
-  sections: {
-    architecture: instructions`
-      Core engine in ${file("src/compile.ts")}.
-      Compile specs with ${symbol("src/compile.ts", "compileClaude")}.
-      Run ${cmd("npm test")} to verify.
-      See ${ref("skills/strengthen/SKILL.md")} for the strengthen skill.
-    `,
-    // If any path / script / symbol is stale → compile error
-  },
-  // ...
-});
-```
+The wizard auto-detects your project, creates a spec, scans your linters, compiles to markdown, adds a CI step, and installs Claude Code hooks. After install: the agent edits the spec (hooks block direct CLAUDE.md edits), the spec auto-compiles on save, and `vigiles audit` catches drift in CI. Prefer no new files? Stay in [markdown mode](docs/markdown-mode.md). Start with `guidance()` rules and `/strengthen` them to `enforce()` later; flags and agent usage are in the [CLI reference](docs/cli.md).
 
-There's a small family of inline **marks** that `audit` checks, each binding a reference to its real source:
-
-- `` `vigiles:symbol file#name` `` — the named file actually **defines** that symbol (function, class, method, constant), parsed with [ast-grep](https://ast-grep.github.io) across **JS/TS, Python, Ruby, Rust, and CSS**. Rename it and `audit` fails; in markdown mode the `refs-hook` **forces the mark**, blocking edits that leave a code reference bare. [Details →](research/symbol-verification.md)
-- `` `vigiles:mcp server#tool` `` — the referenced **MCP tool exists** on its server. `audit` reads `.mcp.json`, starts the server, lists its tools, and flags a renamed/removed one with a "did you mean" — catching e.g. the GitHub MCP server renaming `create_issue` → `issue_write`, which otherwise fails silently.
-
-**Typo-safe at authoring time, too.** `vigiles generate-types` emits a `.vigiles/generated.d.ts` so `enforce("eslint/no-consolee")` red-squiggles in your editor; `generate-schema` gives Level 1 frontmatter the same via your YAML language server. Both have `--check` CI freshness modes. [How it works →](docs/linter-support.md#generate-types)
+Companion repo for [Feedback Loop Is All You Need](https://zernie.com/blog/feedback-loop-is-all-you-need).
 
 ## Test your Claude Code harness
 
-You wrote hooks, a skill, a CLAUDE.md rule — how do you know they work, beyond
-running Claude and eyeballing it? vigiles ships a library to **test the harness
-itself**, at three levels, cheapest first. It's plain async functions, so it
-drops into **node:test / vitest / jest**, or a zero-setup `vigiles test`.
+Verifying references proves your instructions are _true_ — but the hooks and
+skills that enforce them still have to **actually fire**. A hook can be wired
+wrong, a skill's description can fail to trigger, injected context can never reach
+the model — all silently, all passing a naive "did it run?" check. So vigiles's
+second pillar **tests the harness itself**, as the assembled machine it ships as.
 
-### Level 1 — test a hook by itself (no AI, milliseconds)
+> **Set it up — paste this into Claude Code:**
+>
+> > Install vigiles and use its `test-harness` skill to write and run a harness test for this project. If I didn't say what to test, pick something real from my hooks / skills / settings, choose the cheapest tier (unit / deterministic / eval), write the test, and run it. Use good defaults, but **ask me** whether to gate it in CI and whether to add a real-model eval.
 
-A hook is just a process that's handed a "Claude is about to do X" event and
-answers block/allow. Hand it a fake event and check the answer — no `claude`, no
-model, and **every** event type is reachable (incl. Edit/Write, PreCompact,
-SessionEnd):
+It's a small library of plain async functions (drops into node:test / vitest /
+jest, or a zero-setup `vigiles test`), with three tiers, cheapest first. The
+design bet is **deterministic and cheap**: the first two tiers never call a model
+or need an API key, so they run on every commit for free — the opposite of
+eval-only frameworks like promptfoo, where every run hits a real model **by
+design**. You only reach for the paid real-model tier when the question genuinely
+needs it.
+
+- **Unit-test a hook** — `runHook` hands a hook a fake event and checks block/allow. No `claude`, no model, milliseconds, reaches **every** event type.
+- **Deterministic harness test** — `runHarnessTest` runs the **real** `claude` against a **scripted mock model**, so your hooks fire for real with no API key and the same result every time.
+- **Eval** — `runEval` runs the real model A/B (change on vs off) and reports the gap as **mean ± se**, with a Welch-t-test [significance gate](docs/harness-testing.md#significance--is-the-gap-real), regression baselines, and cost/latency/token tracking.
 
 ```typescript
-import { runHook } from "vigiles/run-hook";
+import { runHook } from "vigiles/testing";
 
 const r = runHook(guardCommand, {
   hook_event_name: "PreToolUse",
@@ -270,123 +142,10 @@ const r = runHook(guardCommand, {
 assert(r.blocked); // exit 2 / decision:"block" / permissionDecision:"deny"
 ```
 
-The same shape governs **MCP tools** — the dominant real MCP test — with no
-server running, because the hook only sees the tool _name_:
-
-```typescript
-// block the destructive github-MCP tool; read-only ones pass
-runHook(guard, {
-  hook_event_name: "PreToolUse",
-  tool_name: "mcp__github__merge_pull_request",
-  tool_input: { pull_number: 42 },
-}).blocked; // true
-```
-
-### Level 2 — does it fire in a real session? (free, scripted "AI")
-
-Right logic ≠ wired in correctly. `runHarnessTest` runs the **real** `claude`
-against a **scripted mock model** you control — your hooks fire for real, the
-agent's turns are fixed, no API key, same result every time. Covers the
-governance shapes: SessionStart, Stop, UserPromptSubmit, and Bash **and
-Edit/Write** Pre/PostToolUse.
-
-```typescript
-import { runHarnessTest, scriptModel } from "vigiles/harness-test";
-
-const r = await runHarnessTest({
-  settings: {
-    hooks: {
-      Stop: [
-        { hooks: [{ type: "command", command: "test -f DONE || exit 2" }] },
-      ],
-    },
-  },
-  model: scriptModel([
-    { text: "I'm done" }, // tries to stop → blocked (no DONE)
-    { tool: "Bash", input: { command: "touch DONE" } },
-    { text: "now done" },
-  ]),
-});
-assert(JSON.parse(r.stdout).num_turns > 1); // the Stop hook forced more work
-```
-
-### Level 3 — does it change what Claude does? (real AI, occasional)
-
-`runEval` runs the **real** model N times with your change **on vs off** and
-reports the gap. Costs tokens, so you run it now and then — not on every save:
-
-```typescript
-import { runEval, formatEvalReport } from "vigiles/eval";
-
-const report = await runEval({
-  arms: { off: {}, on: { settings: { hooks: { PostToolUse: [refsHook] } } } },
-  task: "Document chargeCard in SKILL.md, referencing it by name.",
-  measure: (ctx) => ({
-    marked: ctx.sh("grep -c vigiles:symbol SKILL.md") !== "0",
-  }),
-  trials: 6,
-});
-console.log(formatEvalReport(report)); // off marked=0.00   on marked=0.50
-```
-
-### Test your skills for real — and assert on what Claude _did_
-
-Install a plugin the way Claude actually does (`pluginDir` → `--plugin-dir`) so
-its **skills genuinely activate**, then assert on the agent's _actions_, not a
-stdout grep:
-
-```typescript
-import { assertSkillResolved, assertToolNotUsed } from "vigiles/harness-assert";
-
-const r = await runHarnessTest({
-  pluginDir: "./my-plugin",
-  transcript: true, // populate r.toolCalls
-  allowedTools: ["Read", "Write", "Bash", "Skill"],
-  model: scriptModel([
-    { tool: "Skill", input: { skill: "my-plugin:greet" } },
-    { text: "ok" },
-  ]),
-});
-assertSkillResolved(r, "my-plugin:greet"); // the skill fired, no error
-assertToolNotUsed(r, /^mcp__github__merge/); // the safety negative: the scary tool was never called
-```
-
-`assertToolNotUsed` is how you test a safety rule **honestly** — _proving_ the
-dangerous tool was never used, which "the file looks unchanged" can't. It works
-on **real third-party plugins** too: the suite confirms real `obra/superpowers`
-and `wshobson/agents` skills resolve this way, with no markers injected.
-
-### Run them in CI
-
-`vigiles test` runs `*.harness.mjs` files (free, no key); `vigiles eval` runs
-`*.eval.mjs` files (real model). Point a test at a whole plugin (or `"./"` for
-your repo) to load **what ships** — hooks (with `${CLAUDE_PLUGIN_ROOT}`
-resolved), CLAUDE.md, skills, subagents, commands — and `loadPlugin().warnings`
-flags anything only a real model can drive, so you never silently test an empty
-machine.
-
-```bash
-npx vigiles test examples/harness/policy-gate.harness.mjs
-npx vigiles eval --trials=6 examples/harness/skill-outcome.eval.mjs
-```
-
-### What's covered today — surface × tier
-
-| Surface                                                       | Unit / static                | Integration (no API key)    | Eval (real model) |
-| ------------------------------------------------------------- | ---------------------------- | --------------------------- | ----------------- |
-| Hooks — Bash / SessionStart / Stop / UserPromptSubmit         | ✅ logic                     | ✅ fires                    | ✅                |
-| Hooks — Edit / Write                                          | ✅ logic                     | ✅ fires                    | ✅                |
-| Hooks — PreCompact / Notification / SessionEnd / SubagentStop | ✅ logic                     | — (mock can't trigger)      | 🟡                |
-| CLAUDE.md / instructions                                      | ✅ refs                      | 🟡 present, not behaviour   | ✅ behaviour      |
-| Skills                                                        | 🟡 refs                      | ✅ resolves via `pluginDir` | ✅ activation     |
-| Subagents (`agents/`)                                         | 🟡 refs                      | 🔴 hard                     | ✅ via Task       |
-| Slash commands (`commands/`)                                  | 🟡 refs                      | 🟡 needs prompt capture     | ✅ via `/cmd`     |
-| MCP servers                                                   | ✅ tool refs (`vigiles:mcp`) | 🔴                          | 🔴                |
-| settings.json                                                 | 🟡 assert merged             | ✅ applied                  | ✅                |
-
-✅ shipped · 🟡 partial · 🔴 gap · — n/a. Full detail + roadmap: [`research/harness-testing-coverage-matrix.md`](research/harness-testing-coverage-matrix.md).
-
-[Full guide → `docs/harness-testing.md`](docs/harness-testing.md) · [benchmarks](research/benchmarks-runtime-gates.md).
+**[Full guide → `docs/harness-testing.md`](docs/harness-testing.md)** — the tier
+walkthrough, testing skills for real, "fired ≠ landed" (`trace.modelRequests`),
+the safe-by-default sandbox for untrusted plugins, the surface × tier coverage
+matrix, and how it compares to promptfoo. Also: [benchmarks](research/benchmarks-runtime-gates.md).
 
 ## CLI & CI
 
@@ -405,43 +164,29 @@ npx vigiles eval        # Run *.eval.mjs real-model harness evals (--trials=N)
 Install with [Vercel Skills](https://github.com/vercel-labs/skills): `npx skills add zernie/vigiles`
 
 <details>
-<summary><b>The 7 skills</b></summary>
-
-| Skill                  | What it does                                                            |
-| ---------------------- | ----------------------------------------------------------------------- |
-| `strengthen`           | Upgrade `guidance()` → `enforce()` using linter-specific reference docs |
-| `edit-spec`            | Edit a spec file — guided workflow with compile step                    |
-| `migrate-to-spec`      | Convert a hand-written CLAUDE.md to a typed `.spec.ts`                  |
-| `generate-rule`        | Add a new `enforce()` / `guidance()` rule to a spec                     |
-| `pr-to-lint-rule`      | Turn a recurring PR review comment into a lint rule + spec entry        |
-| `enforce-rules-format` | Validate all rules have enforcement classification                      |
-| `audit-feedback-loop`  | Score your repo's feedback loop maturity                                |
-
-</details>
-
-## Maturity Levels
-
-From [Feedback Loop Is All You Need](https://zernie.com/blog/feedback-loop-is-all-you-need): **Vibes → Guardrails → Architecture as Code → The Organism**.
-
-<details>
-<summary>What each level means</summary>
-
-| Level | Name                 | What it means                                                       |
-| ----- | -------------------- | ------------------------------------------------------------------- |
-| 0     | Vibes                | No CI, no linters, no CLAUDE.md                                     |
-| 1     | Guardrails           | CI + standard linters, no custom rules                              |
-| 2     | Architecture as Code | Custom lint rules + enforced CLAUDE.md                              |
-| 3     | The Organism         | CI + custom rules + visual tests + observability + scheduled agents |
+<summary><b>The 8 skills</b></summary>
+
+| Skill                  | What it does                                                                                      |
+| ---------------------- | ------------------------------------------------------------------------------------------------- |
+| `strengthen`           | Upgrade `guidance()` → `enforce()` using linter-specific reference docs                           |
+| `edit-spec`            | Edit a spec file — guided workflow with compile step                                              |
+| `migrate-to-spec`      | Convert a hand-written CLAUDE.md to a typed `.spec.ts`                                            |
+| `generate-rule`        | Add a new `enforce()` / `guidance()` rule to a spec                                               |
+| `pr-to-lint-rule`      | Turn a recurring PR review comment into a lint rule + spec entry                                  |
+| `enforce-rules-format` | Validate all rules have enforcement classification                                                |
+| `audit-feedback-loop`  | Score your repo's feedback loop maturity                                                          |
+| `test-harness`         | Test a Claude Code harness — pick the tier (unit / deterministic / eval) and write a passing test |
 
 </details>
 
 ## Related Tools
 
-vigiles owns one thing: compile-time verification of typed specs against real linter configs, filesystems, and package.json, plus testing the harness those specs describe. Everything else it composes with rather than replaces — architectural linters ([ast-grep](https://ast-grep.github.io/), [Dependency Cruiser](https://github.com/sverweij/dependency-cruiser)) referenced via `enforce()`, file-sync tools ([Ruler](https://github.com/intellectronica/ruler), [rulesync](https://github.com/dyoshikawa/rulesync)) that distribute the compiled output, and markdown/prose linters that check a different layer. [How vigiles composes with each, and why runtime-LLM rule checkers are the opposite paradigm →](docs/related-tools.md)
+vigiles composes with other tools rather than replacing them: architectural linters ([ast-grep](https://ast-grep.github.io/), [Dependency Cruiser](https://github.com/sverweij/dependency-cruiser)) referenced via `enforce()`, and file-sync tools ([Ruler](https://github.com/intellectronica/ruler), [rulesync](https://github.com/dyoshikawa/rulesync)) that distribute the compiled output. [How it fits with each, and why runtime-LLM rule checkers are the opposite paradigm →](docs/related-tools.md)
 
 ## Documentation
 
-- **[docs/](docs/README.md)** — how-to & reference: the adoption ladder, CLI, linter support, the harness-testing guide, skills/agents.
+- **The two pillar guides:** [verifying instruction files](docs/verifying-instruction-files.md) (Pillar 1) · [testing your harness](docs/harness-testing.md) (Pillar 2).
+- **[docs/](docs/README.md)** — the full how-to & reference index: adoption ladder, CLI, linter support, skills/agents.
 - **[research/](research/README.md)** — the thinking behind it: design docs, the [harness-testing coverage roadmap](research/harness-testing-coverage-matrix.md), benchmark findings, landscape, and parked ideas.
 
 ## License

package/dist/agent-result.d.ts

@@ -0,0 +1,40 @@
+/**
+ * vigiles — parse a subagent's railway result.
+ *
+ * A subagent with a `result()` contract is told (in its compiled system prompt)
+ * to end its turn with exactly one fenced block:
+ *
+ *   ```vigiles:ok
+ *   { "files": ["a.ts"], "summary": "done" }
+ *   ```
+ *
+ * or `vigiles:err` for the error track. This module extracts and validates that
+ * block — the single primitive the railway orchestrator and the harness-test
+ * assertions (`assertAgentOk`/`assertAgentErr`) both build on. Pure and
+ * model-free: hand it the worker's text, get back a discriminated outcome.
+ *
+ * "Railway-oriented" is literal here: the parse is `text -> Result<S, E>` with a
+ * third `malformed` track for a worker that didn't honor its contract (no block,
+ * bad JSON, or a shape that doesn't match the declared schema).
+ */
+import type { OutputContract } from "./spec.js";
+/** The outcome of parsing a worker's result block. */
+export type ParsedAgentResult<S = Record<string, unknown>, E = Record<string, unknown>> = {
+    readonly kind: "ok";
+    readonly value: S;
+} | {
+    readonly kind: "err";
+    readonly error: E;
+} | {
+    readonly kind: "malformed";
+    readonly reason: string;
+};
+/**
+ * Parse the last `vigiles:ok` / `vigiles:err` block from a worker's output.
+ *
+ * With a `contract`, the parsed object is validated against the matching track's
+ * shape — a worker that emits the wrong shape is `malformed`, not a silent pass.
+ * Without one, any well-formed JSON block is accepted.
+ */
+export declare function parseAgentResult(text: string, contract?: OutputContract): ParsedAgentResult;
+//# sourceMappingURL=agent-result.d.ts.map

package/dist/agent-result.js

@@ -0,0 +1,97 @@
+"use strict";
+/**
+ * vigiles — parse a subagent's railway result.
+ *
+ * A subagent with a `result()` contract is told (in its compiled system prompt)
+ * to end its turn with exactly one fenced block:
+ *
+ *   ```vigiles:ok
+ *   { "files": ["a.ts"], "summary": "done" }
+ *   ```
+ *
+ * or `vigiles:err` for the error track. This module extracts and validates that
+ * block — the single primitive the railway orchestrator and the harness-test
+ * assertions (`assertAgentOk`/`assertAgentErr`) both build on. Pure and
+ * model-free: hand it the worker's text, get back a discriminated outcome.
+ *
+ * "Railway-oriented" is literal here: the parse is `text -> Result<S, E>` with a
+ * third `malformed` track for a worker that didn't honor its contract (no block,
+ * bad JSON, or a shape that doesn't match the declared schema).
+ */
+Object.defineProperty(exports, "__esModule", { value: true });
+exports.parseAgentResult = parseAgentResult;
+// Capture every vigiles:ok / vigiles:err fenced block; the LAST one is the
+// worker's final answer (earlier ones may be illustrative in its reasoning).
+const BLOCK_RE = /```vigiles:(ok|err)[ \t]*\r?\n([\s\S]*?)```/g;
+/** Does a runtime value match a declared field type? */
+function fieldMatches(value, type) {
+    switch (type) {
+        case "string":
+            return typeof value === "string";
+        case "number":
+            return typeof value === "number";
+        case "boolean":
+            return typeof value === "boolean";
+        case "string[]":
+            return Array.isArray(value) && value.every((v) => typeof v === "string");
+    }
+}
+/** Validate a parsed object against a contract track; null when it conforms. */
+function shapeError(obj, shape) {
+    for (const [field, type] of Object.entries(shape)) {
+        if (!(field in obj))
+            return `missing field "${field}"`;
+        if (!fieldMatches(obj[field], type)) {
+            return `field "${field}" should be ${type}`;
+        }
+    }
+    return null;
+}
+/**
+ * Parse the last `vigiles:ok` / `vigiles:err` block from a worker's output.
+ *
+ * With a `contract`, the parsed object is validated against the matching track's
+ * shape — a worker that emits the wrong shape is `malformed`, not a silent pass.
+ * Without one, any well-formed JSON block is accepted.
+ */
+function parseAgentResult(text, contract) {
+    BLOCK_RE.lastIndex = 0;
+    let last = null;
+    for (let m = BLOCK_RE.exec(text); m !== null; m = BLOCK_RE.exec(text)) {
+        last = { track: m[1], body: m[2] };
+    }
+    if (!last) {
+        return {
+            kind: "malformed",
+            reason: "no vigiles:ok/vigiles:err block found",
+        };
+    }
+    let parsed;
+    try {
+        parsed = JSON.parse(last.body);
+    }
+    catch {
+        return {
+            kind: "malformed",
+            reason: `invalid JSON in vigiles:${last.track} block`,
+        };
+    }
+    if (typeof parsed !== "object" || parsed === null || Array.isArray(parsed)) {
+        return {
+            kind: "malformed",
+            reason: `vigiles:${last.track} block must be a JSON object`,
+        };
+    }
+    const obj = parsed;
+    if (contract) {
+        const shape = last.track === "ok" ? contract.ok : contract.err;
+        const err = shapeError(obj, shape);
+        if (err) {
+            return { kind: "malformed", reason: `${last.track} block: ${err}` };
+        }
+    }
+    return last.track === "ok"
+        ? { kind: "ok", value: obj }
+        : { kind: "err", error: obj };
+}
+//# sourceMappingURL=agent-result.js.map