vigiles 5.0.0 → 5.0.1

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (2) hide show
  1. package/README.md +71 -111
  2. package/package.json +27 -4
package/README.md CHANGED
@@ -5,12 +5,11 @@
5
5
  <h1 align="center">vigiles</h1>
6
6
 
7
7
  <p align="center">
8
- <em>Quis custodiet ipsos custodes?</em> — Who watches the watchmen?
8
+ <strong>Lint &amp; test the harness your AI agent runs on.</strong>
9
9
  </p>
10
10
 
11
11
  <p align="center">
12
- <strong>The missing linting + testing layer for agentic coding.</strong><br />
13
- vigiles <strong>lints</strong> the references your instruction files make — linter rules, file paths, scripts, code symbols — and <strong>tests</strong> whether your hooks, skills, and CLAUDE.md actually change what the agent does.
12
+ Your CLAUDE.md, hooks, and skills steer the agent — but nothing checks they're <em>true</em>, and nothing tests they <em>work</em>. vigiles does both.
14
13
  </p>
15
14
 
16
15
  <p align="center">
@@ -22,113 +21,17 @@
22
21
  ---
23
22
 
24
23
  `Agent = Model + Harness`. You'd never ship an app without a linter and a test
25
- suite — yet an AI agent steering your repo is trusted on vibes. vigiles is the
26
- deterministic layer for the harness: it **lints** the references your instruction
27
- files make and **tests** that your hooks and skills actually fire. Two independent
28
- pillars — adopt either, or both:
24
+ suite — yet the harness steering your agent runs on vibes. vigiles[^name] is the
25
+ deterministic layer for it, and does two independent things — adopt either, or both:
29
26
 
30
- | | Pillar | What it does |
31
- | ----- | ------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
32
- | **①** | **Lint your instruction files** | Every linter rule, file path, script, and code symbol your CLAUDE.md cites is checked against reality, so stale references can't silently mislead the agent. → [guide](docs/verifying-instruction-files.md) |
33
- | **②** | **Test your harness** | Your hooks and skills are code — vigiles tests they actually fire, **deterministically and free** (no model, no API key); and when a question _does_ need a real-model eval, it runs on your **Claude subscription**, not metered API. → [guide](docs/harness-testing.md) |
27
+ | | |
28
+ | ----------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
29
+ | **🔎 Lint** | Every file path, script, code symbol, and linter rule your CLAUDE.md cites is checked against reality — so a renamed file or a disabled rule can't silently mislead the agent. **[→](docs/verifying-instruction-files.md)** |
30
+ | **🧪 Test** | Hooks, skills, and subagents are code. vigiles tests they _do their job_ — and almost all of it is **deterministic, no API key**; the real-model evals run on your **Claude subscription**, not metered tokens. **[→](docs/harness-testing.md)** |
34
31
 
35
- Neither pillar depends on the other — pick the one that hurts today. **Works with
36
- Claude Code and Codex** ([`vigiles/codex`](docs/harnesses.md)) behind a five-port
37
- adapter; [custom adapters welcome](docs/authoring-an-adapter.md).
38
-
39
- ## ① Lint — your CLAUDE.md lies to your agent
40
-
41
- Your CLAUDE.md says _"enforce `eslint/no-console`."_ But it was switched off
42
- months ago — and the agent trusts the claim. (Same story for the file path it
43
- cites that got renamed, and the script that was deleted.)
44
-
45
- **Without vigiles:** nobody checks. The agent acts on fiction.
46
-
47
- **With vigiles:** `npx vigiles lint` resolves every reference against reality —
48
-
49
- ```text
50
- CLAUDE.md (inline mode):
51
- ✗ line 1: Rule "eslint/no-console" exists but is disabled in eslint config
52
- ✓ line 2: eslint/eqeqeq
53
- ✗ line 3: Rule "no-consoel" not found in eslint. Did you mean: "eslint/no-console"?
54
- ```
55
-
56
- It resolves rule names across **7 linter catalogs** — the rule exists **and is
57
- enabled** — and checks file paths, scripts, and code symbols the same way. Start
58
- with one comment, no new files:
59
-
60
- ```md
61
- <!-- vigiles:enforce eslint/no-console "Route output through logger.ts" -->
62
- ```
63
-
64
- Step up to a typed `.spec.ts` (compiled to CLAUDE.md, compiler-grade) when you
65
- want it. **[Full guide →](docs/verifying-instruction-files.md)**
66
-
67
- ## ② Test — does your harness actually fire?
68
-
69
- A hook can be wired wrong, a skill's description can fail to trigger, injected
70
- context can never reach the model — silently, all passing a naive "did it run?"
71
- check.
72
-
73
- **Without vigiles:** you assume your `--no-verify` guard blocks. You don't know.
74
-
75
- **With vigiles:** a deterministic test proves it — no model, no API key,
76
- milliseconds:
77
-
78
- ```typescript
79
- import { runHook } from "vigiles/testing";
80
-
81
- const r = runHook(guard, {
82
- hook_event_name: "PreToolUse",
83
- tool_name: "Bash",
84
- tool_input: { command: "git commit --no-verify" },
85
- });
86
- assert(r.blocked); // a red ✗ here means your hook silently lets it through
87
- ```
88
-
89
- ```text
90
- ✓ guard blocks --no-verify and allows a clean commit
91
-
92
- 2 passed.
93
- ```
94
-
95
- Three tiers, cheapest first: **`runHook`** (a hook's logic), **`runHarnessTest`**
96
- (the real agent CLI against a scripted mock model), and the real-model scored tier
97
- (**`measure`** / **`runEval`**). **Testing a skill?** Two questions, both covered:
98
- does its description **fire** (`measureTriggerRate` — recall across varied prompts
99
- without hijacking unrelated ones, precision), **and** does its guidance actually
100
- **work**. For "is this exact skill any good?" score the output directly —
101
- `measure({ checks: [judged(rubric)] })` + `assertRates` (the **absolute** oracle,
102
- what promptfoo/DeepEval lead with; no on/off baseline needed). When you need the
103
- **relative** lift over no-skill — regression, or proving the change isn't noise —
104
- A/B it on-vs-off with `runEval` + `assertSignificant`. Description _and_ behavior,
105
- not just one. **Need a safety property** — that the agent
106
- **didn't** push to the wrong branch or call a paid API? `notTool` + `interceptTools`
107
- intercept the tool in the real hook layer, so the attempt is caught and the side
108
- effect never happens. **[Full guide →](docs/harness-testing.md)** · vigiles runs
109
- foreign code (and a real model) safely by default — **[safety model →](docs/safety.md)**
110
-
111
- Most of what real plugins do is testable cheaply — fire / trigger / contract /
112
- safety, plus **record-replay** for the tool/API results a skill consumes (recorded
113
- once from the real tool, replayed deterministically — no live service, no Docker).
114
- That covers ~90%+ of real plugin surface on your subscription; the rare case that
115
- needs a real browser or database **composes with Docker** rather than us
116
- reinventing the sandbox. **[What we test, how →](research/eval-coverage-and-isolation.md)**
117
-
118
- **Affordable by design — the eval you can actually run.** Almost nobody evals
119
- their harness, because the usual tools (promptfoo, DeepEval, …) hit the API SDK
120
- and bill **per token on every run**. vigiles inverts that: most questions are
121
- answered with **no model at all** (free, every commit), and when you do reach for
122
- a real-model eval, vigiles drives your `claude` CLI — so it runs on the **Pro/Max
123
- subscription you already pay for**, not metered API billing. CI runs only the free
124
- deterministic tiers; you run the real-model eval where the subscription already is
125
- — a Claude Code session or locally — when it's worth it, not on every PR.
126
-
127
- This affordability story is **ToS-clean**: vigiles drives _your own_ `claude` CLI
128
- to test _your own_ harness on _your own_ subscription — the same thing you do when
129
- you run Claude Code. (The Claude Agent SDK's ToS restricts _productizing_ claude.ai
130
- login/limits in a third-party offering; running your own tests on your own sub is
131
- exactly the supported posture, not that.)
32
+ Pick the one that hurts today. **Works with Claude Code and Codex**
33
+ ([`vigiles/codex`](docs/harnesses.md)) behind a five-port adapter;
34
+ [custom adapters welcome](docs/authoring-an-adapter.md).
132
35
 
133
36
  ## Quick start
134
37
 
@@ -137,7 +40,7 @@ exactly the supported posture, not that.)
137
40
  ```text
138
41
  Install vigiles in this repo and run it. Verify my CLAUDE.md / AGENTS.md
139
42
  references and show me what's stale, then write and run a harness test for one
140
- of my hooks or skills. Use good defaults (both pillars, non-interactive), but
43
+ of my hooks or skills. Use good defaults (lint + test, non-interactive), but
141
44
  ask me first whether to gate it in CI, whether to add a real-model eval, and
142
45
  whether to enforce strictly (--strict).
143
46
  ```
@@ -145,7 +48,7 @@ whether to enforce strictly (--strict).
145
48
  Or do it yourself:
146
49
 
147
50
  ```bash
148
- npx vigiles init # sets up BOTH pillars: spec + harness test + CI + plugin
51
+ npx vigiles init # sets up lint + test: spec + harness test + CI + plugin
149
52
  ```
150
53
 
151
54
  It's interactive in a terminal and non-interactive for agents/CI (or with
@@ -156,7 +59,7 @@ tell your agent _"test my skills"_ and it picks the tier and writes the test.
156
59
  <details>
157
60
  <summary>What <code>init</code> sets up</summary>
158
61
 
159
- - **Both pillars** by default; scope with `--lint` / `--test` (one or both).
62
+ - **Both lint and test** by default; scope with `--lint` / `--test` (one or both).
160
63
  - Adds `vigiles` to your `devDependencies`.
161
64
  - Installs the Claude Code plugin (skills + hooks) via the marketplace —
162
65
  globally, never vendored into your repo.
@@ -172,6 +75,61 @@ Prefer to write tests yourself? They can be JS **or** TS
172
75
 
173
76
  </details>
174
77
 
78
+ ## ① Lint — your CLAUDE.md lies to your agent
79
+
80
+ Your CLAUDE.md points the agent at `src/auth/login.ts` and tells it to run
81
+ `npm run check`. But the file moved to `src/auth/session.ts` six commits ago, and
82
+ the script was renamed. The agent trusts the stale claim and acts on fiction.
83
+
84
+ `npx vigiles lint` resolves every reference against reality:
85
+
86
+ ```text
87
+ CLAUDE.md:
88
+ ✗ src/auth/login.ts — no such file (renamed or moved?)
89
+ ✗ npm run check — not in package.json. Did you mean: "check:types"?
90
+ ✓ @typescript-eslint/no-floating-promises — exists and enabled in eslint config
91
+ ```
92
+
93
+ File paths, scripts, and code symbols — plus linter rules across **7 catalogs**
94
+ (the rule exists **and is enabled**). Start with one inline comment, no new files;
95
+ step up to a typed `.spec.ts` (compiled to CLAUDE.md, compiler-grade) when you want
96
+ it. **[Full guide →](docs/verifying-instruction-files.md)**
97
+
98
+ ## ② Test — does your harness do its job?
99
+
100
+ A hook can be wired wrong. A skill's description can fail to trigger — or hijack
101
+ unrelated prompts. Injected context can never reach the model. All of it passes a
102
+ naive "did it run?" check. vigiles tests the assembled harness for real:
103
+
104
+ ```typescript
105
+ import { runHook } from "vigiles/testing";
106
+
107
+ const r = runHook(guard, {
108
+ hook_event_name: "PreToolUse",
109
+ tool_name: "Bash",
110
+ tool_input: { command: "git commit --no-verify" },
111
+ });
112
+ assert(r.blocked); // a red ✗ means your guard silently lets it through
113
+ ```
114
+
115
+ It goes well past _"did it fire?"_:
116
+
117
+ - **Hooks block** what they must — `runHook`, or the real agent CLI via `runHarnessTest`.
118
+ - **Skills trigger** on the right prompts and stay quiet on the wrong ones — recall _and_ precision (`measureTriggerRate`).
119
+ - **Behaviour is good** — score a skill's output directly, or A/B it on-vs-off for the real lift over no-skill (`measure` / `runEval`, with significance testing).
120
+ - **Safety holds** — the agent _didn't_ push to the wrong branch or hit a paid API; `interceptTools` catches the attempt so the side effect never happens.
121
+
122
+ **The eval you can actually afford.** Almost every tier runs with **no model and
123
+ no API key** — milliseconds, on every commit. The rest drive your own `claude` CLI:
124
+
125
+ | | Runs on | Cost |
126
+ | ---------------------- | ----------------------- | ------------------------------------------- |
127
+ | promptfoo, DeepEval, … | metered API SDK | billed **per token, every run** |
128
+ | **vigiles** | your Claude Pro/Max sub | **$0 extra** — and most tiers need no model |
129
+
130
+ That's why you can eval your harness on every change, not just once.
131
+ **[How it works →](docs/harness-testing.md)** · **[Why it's affordable →](docs/eval-architecture.md)** · **[Safety model →](docs/safety.md)**
132
+
175
133
  ## More
176
134
 
177
135
  - **[CLI & GitHub Action →](docs/cli.md)** — every command, the Action (inputs / output / versioning), the Claude Code plugin, and the five `lint` rules.
@@ -182,3 +140,5 @@ Prefer to write tests yourself? They can be JS **or** TS
182
140
  ## License
183
141
 
184
142
  [MIT](LICENSE)
143
+
144
+ [^name]: **vigiles** — the watchmen of ancient Rome, who guarded the city (and fought its fires) by night. _Quis custodiet ipsos custodes?_ — "who watches the watchmen?" (Juvenal, _Satire VI_).
package/package.json CHANGED
@@ -1,7 +1,32 @@
1
1
  {
2
2
  "name": "vigiles",
3
- "version": "5.0.0",
4
- "description": "Compile .spec.ts files to instruction files (CLAUDE.md, AGENTS.md) with linter cross-referencing",
3
+ "version": "5.0.1",
4
+ "description": "Lint & test the harness your AI agent runs on — verify the references in your CLAUDE.md / AGENTS.md and test that your hooks and skills actually work.",
5
+ "keywords": [
6
+ "claude-code",
7
+ "codex",
8
+ "agents",
9
+ "agentic",
10
+ "ai",
11
+ "llm",
12
+ "harness",
13
+ "hooks",
14
+ "skills",
15
+ "claude",
16
+ "agents-md",
17
+ "eval",
18
+ "testing",
19
+ "linter"
20
+ ],
21
+ "homepage": "https://github.com/zernie/vigiles#readme",
22
+ "repository": {
23
+ "type": "git",
24
+ "url": "git+https://github.com/zernie/vigiles.git"
25
+ },
26
+ "bugs": {
27
+ "url": "https://github.com/zernie/vigiles/issues"
28
+ },
29
+ "license": "MIT",
5
30
  "bin": {
6
31
  "vigiles": "dist/cli.js"
7
32
  },
@@ -46,7 +71,6 @@
46
71
  "lint": "eslint src/",
47
72
  "fmt": "prettier --write .",
48
73
  "fmt:check": "prettier --check .",
49
- "demo": "npm run build && bash examples/demo/run.sh",
50
74
  "test:unit": "npm run build && vitest run --project unit",
51
75
  "test:integration": "npm run build && vitest run --project integration",
52
76
  "test:e2e": "npm run build && vitest run --project e2e",
@@ -56,7 +80,6 @@
56
80
  "test:vitest": "npm run build && vitest run --project runners",
57
81
  "test:jest": "npm run build && jest",
58
82
  "test:types": "npm run build && tsc --noEmit -p test/types/tsconfig.json",
59
- "demo:plugin": "npm run build && node examples/plugin-test-demo.mjs",
60
83
  "api:report": "npm run build && node scripts/api-extractor.mjs --local",
61
84
  "api:check": "npm run build && node scripts/api-extractor.mjs",
62
85
  "docs:api": "npm run api:report && api-documenter markdown -i temp -o api-reference"