vigiles 9.0.0 → 10.0.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +144 -222
- package/dist/audit-html.d.ts +15 -4
- package/dist/audit-html.js +15 -6
- package/dist/audit-report.d.ts +94 -1
- package/dist/audit-report.js +50 -0
- package/dist/audit-report.template.html +47 -22
- package/dist/audit-score.d.ts +27 -7
- package/dist/audit-score.js +63 -50
- package/dist/audit-serve.d.ts +109 -0
- package/dist/audit-serve.js +257 -0
- package/dist/cli.js +322 -25
- package/dist/core/adopt.d.ts +28 -0
- package/dist/core/adopt.js +203 -0
- package/dist/core/compile.d.ts +5 -1
- package/dist/core/compile.js +19 -10
- package/dist/leaderboard.d.ts +32 -0
- package/dist/leaderboard.js +109 -50
- package/dist/scan-behavioral.d.ts +85 -0
- package/dist/scan-behavioral.js +225 -0
- package/dist/scan.js +23 -1
- package/package.json +1 -1
package/README.md
CHANGED
|
@@ -3,42 +3,55 @@
|
|
|
3
3
|
This file is the FRONT DOOR + a marketing asset for someone who already lives
|
|
4
4
|
in Claude Code / Codex. Optimize for a phone-skimmer.
|
|
5
5
|
|
|
6
|
-
|
|
7
|
-
|
|
8
|
-
|
|
9
|
-
|
|
10
|
-
|
|
11
|
-
|
|
12
|
-
|
|
13
|
-
|
|
14
|
-
|
|
15
|
-
|
|
16
|
-
|
|
17
|
-
|
|
18
|
-
|
|
19
|
-
|
|
20
|
-
|
|
21
|
-
|
|
22
|
-
|
|
23
|
-
|
|
24
|
-
|
|
25
|
-
|
|
26
|
-
|
|
27
|
-
|
|
28
|
-
|
|
29
|
-
|
|
30
|
-
|
|
31
|
-
|
|
32
|
-
|
|
33
|
-
|
|
34
|
-
|
|
35
|
-
|
|
36
|
-
|
|
37
|
-
|
|
38
|
-
|
|
39
|
-
|
|
40
|
-
|
|
41
|
-
|
|
6
|
+
SPINE = CONCEPT 5 (proof/demo-led). Lead with REAL, screenshotable catches on
|
|
7
|
+
plugins people actually ship, THEN explain the mechanism. The proofs are not
|
|
8
|
+
illustrative — every block traces to a real dogfood run captured in
|
|
9
|
+
research/dogfood/. TWO COMMUNITY catches, anonymized (a skill-description collision
|
|
10
|
+
→ wrong-skill-fires (claude-flow, Triggering F), an AskUserQuestion-never-available
|
|
11
|
+
tool) — both real GRADED defects that REPRODUCE on current main. NEVER replace a
|
|
12
|
+
real catch with a fabricated one. (Proof 1 was a missing-SKILL.md/Truthfulness
|
|
13
|
+
catch, swapped 2026-06-28: its source (superpowers) is clean on current main and NO
|
|
14
|
+
reproducible dead-file-ref exists in popular OSS — those are an adopt+strengthen
|
|
15
|
+
payoff, see research/oss-audit-render-findings.md.)
|
|
16
|
+
|
|
17
|
+
WHY ONLY TWO (decided 2026-06-28): the earlier Proofs 3-4 leaned on
|
|
18
|
+
pr-review-toolkit's "review agents inherit all tools" as an official-plugin
|
|
19
|
+
defect. But inherit-all (a subagent with no `tools:` line) is now ADVISORY, not a
|
|
20
|
+
graded penalty — omitting the tool contract is a near-universal, legitimate
|
|
21
|
+
authoring style (an OSS sweep of 122 plugins found 109 whose only finding was
|
|
22
|
+
this), so penalizing it cried wolf. With that change the official plugins are all
|
|
23
|
+
a clean A, so a "even Anthropic has bugs" proof would be dishonest — Proofs 3-4
|
|
24
|
+
were DROPPED rather than reframed. The leaderboard feature still exists; it just
|
|
25
|
+
isn't a headline proof.
|
|
26
|
+
|
|
27
|
+
DON'T SHAME OSS: community catches are real but ANONYMIZED in public copy (no
|
|
28
|
+
obra/superpowers, madappgang by name) — real names live only in research/dogfood/.
|
|
29
|
+
If an official/vendor proof returns, punch UP (name Anthropic's own); never name a
|
|
30
|
+
volunteer's repo to show its bug.
|
|
31
|
+
|
|
32
|
+
1. LEAD WITH BENEFITS / the reader's CONCRETE PAIN, never an apology, caveat, or
|
|
33
|
+
competitor. A bolded lead-in is the first thing read — make it the hook/win.
|
|
34
|
+
End a section on the win, not the trade-off. A paragraph is ≤ ~3 lines.
|
|
35
|
+
2. PROOF FIRST, mechanism second. The three instruments (Lint/Test/Eval) come
|
|
36
|
+
AFTER the proof stack as "how it does it", not as a competing front door.
|
|
37
|
+
3. SPEC-FIRST IS THE DEFAULT but easy — `init` ADOPTS your CLAUDE.md into a spec,
|
|
38
|
+
skills edit it, you rarely hand-write .spec.ts. Give it ONE home (Quick start),
|
|
39
|
+
not five scattered mentions. `eject` always reverses. Inline markdown is the
|
|
40
|
+
zero-TS floor.
|
|
41
|
+
4. Guard / compiled hooks is PARKED FOR LAUNCH (see research/roadmap.md). Live set
|
|
42
|
+
is Lint/Test/Eval. Do NOT make the 2/7→7/7 battery the hero — re-add post-HN.
|
|
43
|
+
5. SCANNABLE + SHORT — ~200-line cap; punchy cells, bullets, runnable blocks.
|
|
44
|
+
Push depth into docs/ and LINK it.
|
|
45
|
+
6. NO INTERNAL VOCABULARY (moat / measurement-authority / flywheel) and NO
|
|
46
|
+
research/ links — name the user benefit.
|
|
47
|
+
7. ASSETS: the hero vigiles-audit.png is a REAL current report (a community
|
|
48
|
+
plugin rendered as "my-plugin" to anonymize) — A 92 with four rings and an
|
|
49
|
+
inline subagent-tool-contract fix; no dialect-drift banner (HTML report is
|
|
50
|
+
terminal-banner-free by design). Re-render via headless Chromium on the React
|
|
51
|
+
report if the UI changes. (vigiles-demo.gif was removed
|
|
52
|
+
from Proof 1 — it rendered as a frozen half-typed terminal and was redundant
|
|
53
|
+
with the code block; if a lint demo returns, it belongs in the Lint section
|
|
54
|
+
with a non-frozen asset.)
|
|
42
55
|
-->
|
|
43
56
|
|
|
44
57
|
<p align="center">
|
|
@@ -48,7 +61,7 @@
|
|
|
48
61
|
<h1 align="center">vigiles</h1>
|
|
49
62
|
|
|
50
63
|
<p align="center">
|
|
51
|
-
<strong>
|
|
64
|
+
<strong>The tests your AI agent harness never had.</strong>
|
|
52
65
|
</p>
|
|
53
66
|
|
|
54
67
|
<p align="center">
|
|
@@ -60,148 +73,80 @@
|
|
|
60
73
|
---
|
|
61
74
|
|
|
62
75
|
**You installed a bunch of plugins and wrote a few skills — but do they actually work?**
|
|
63
|
-
A skill that never fires,
|
|
64
|
-
|
|
76
|
+
A skill that never fires, two skills the agent can't tell apart, a subagent wired to
|
|
77
|
+
a tool that doesn't exist — your harness fails **silently**, and you find out mid-task.
|
|
65
78
|
|
|
66
|
-
**It's a library with no tests.**
|
|
67
|
-
|
|
68
|
-
**One command shows you — like a Lighthouse report for your harness:**
|
|
79
|
+
**It's a library with no tests.** One command runs them — no key, no config, safe on
|
|
80
|
+
any repo:
|
|
69
81
|
|
|
70
82
|
```bash
|
|
71
|
-
npx vigiles audit
|
|
83
|
+
npx vigiles audit
|
|
72
84
|
```
|
|
73
85
|
|
|
74
|
-
|
|
75
|
-
|
|
86
|
+
Here's what it found on real, popular plugins. ↓
|
|
87
|
+
|
|
88
|
+
## The report
|
|
76
89
|
|
|
77
90
|
<p align="center">
|
|
78
|
-
<img src="vigiles-audit.png" width="760" alt="vigiles audit report: an overall score with four category rings — Truthfulness, Triggering, Structure, Tested — and fix
|
|
91
|
+
<img src="vigiles-audit.png" width="760" alt="vigiles audit report: an overall A (92/100) score with four category rings — Truthfulness, Triggering, Structure, Tested — and an inline fix card for a subagent declaring a tool that doesn't exist" />
|
|
79
92
|
</p>
|
|
80
93
|
|
|
81
|
-
|
|
82
|
-
|
|
83
|
-
|
|
84
|
-
|
|
85
|
-
| **🔧 Structure** | Tool contracts, MCP servers & frontmatter are sound |
|
|
86
|
-
| **🧪 Tested** | Every surface ships a test |
|
|
87
|
-
|
|
88
|
-
Like Lighthouse, `audit` is a **local report you run on your machine** — a
|
|
89
|
-
deterministic read, safe on any repo (even one wired to prod), identical on every
|
|
90
|
-
OS. **Not a CI step** (CI uses `lint`, the deterministic gate). The **executing
|
|
91
|
-
checks** (resolve MCP live · **do your skills fire?** · **what would vigiles
|
|
92
|
-
catch in your repo?**) run only when there's a human to consent: `audit` **asks
|
|
93
|
-
once** at a terminal and remembers. **[Audit a harness →](docs/for-plugin-authors.md)**
|
|
94
|
-
|
|
95
|
-
`Agent = Model + Harness` — the model gets the headlines, the harness is the half you
|
|
96
|
-
own. vigiles[^name] is how you make it prove itself: `audit` is the dashboard, and
|
|
97
|
-
**three instruments** fix and prove what it finds —
|
|
98
|
-
|
|
99
|
-
| | |
|
|
100
|
-
| ----------- | ----------------------------------------------------------------------------------------------------------------------------------------------- |
|
|
101
|
-
| **🔎 Lint** | Your CLAUDE.md stops lying — every path, script, symbol & linter rule checked against **reality**. **[→](docs/verifying-instruction-files.md)** |
|
|
102
|
-
| **🧪 Test** | Prove your hooks, skills & subagents do their job — **free, no API key**. **[→](docs/harness-testing.md)** |
|
|
103
|
-
| **📊 Eval** | Know if a skill helps or just costs — **A/B on real tasks**, on your own subscription. **[→](docs/measuring-skills.md)** |
|
|
104
|
-
|
|
105
|
-
<!-- PARKED FOR LAUNCH — Guard / compiled hooks. Re-add this row + the ④ section below post-HN. See research/roadmap.md "Launch readiness".
|
|
106
|
-
| **🛡 Guard** | A safety hook that **can't silently fail open** — write a typed function, get a guard that blocks. **[→](docs/compiled-hooks.md)** |
|
|
107
|
-
-->
|
|
108
|
-
|
|
109
|
-
**Two ways in** — pick the pain that's yours:
|
|
110
|
-
|
|
111
|
-
- **Run agents on your own repo?** `npx vigiles audit`, then `npx vigiles init`.
|
|
112
|
-
- **Ship plugins to a marketplace?** `npx vigiles audit ./plugins/*/` ranks a whole
|
|
113
|
-
marketplace (0–100, A–F) — see the **[plugin-author guide →](docs/for-plugin-authors.md)**.
|
|
94
|
+
Four deterministic rings, **each finding's fix inline**, and a shareable HTML report.
|
|
95
|
+
Like Lighthouse, `audit` is a **local report you run on your machine** — safe on any
|
|
96
|
+
repo (even one wired to prod), identical on every OS. **Not a CI step** (CI uses
|
|
97
|
+
`lint`). **[Audit a harness →](docs/for-plugin-authors.md)**
|
|
114
98
|
|
|
115
|
-
|
|
116
|
-
`.spec.ts` for you, **`init` adopts an existing CLAUDE.md non-destructively**
|
|
117
|
-
(untouched until you compile), and plain markdown + inline `<!-- vigiles:enforce -->`
|
|
118
|
-
comments work with zero TypeScript. Works with **Claude Code and Codex**
|
|
119
|
-
([`vigiles/codex`](docs/harnesses.md)) or [your own harness](docs/authoring-an-adapter.md).
|
|
120
|
-
|
|
121
|
-
## Quick start
|
|
122
|
-
|
|
123
|
-
**Paste into Claude Code or Codex:**
|
|
99
|
+
## Proof 1 — two skills your agent can't tell apart
|
|
124
100
|
|
|
125
101
|
```text
|
|
126
|
-
|
|
127
|
-
|
|
128
|
-
|
|
129
|
-
test for a hook or skill of mine. Don't enforce a spec-per-file or add a real-model
|
|
130
|
-
eval without asking me first.
|
|
102
|
+
✗ Triggering 0
|
|
103
|
+
└ 45 near-identical skill descriptions — the selector can't tell them apart,
|
|
104
|
+
so the wrong one fires (e.g. "agent-coder" ↔ "agent-tester", 83% alike)
|
|
131
105
|
```
|
|
132
106
|
|
|
133
|
-
|
|
107
|
+
A real, popular plugin ships **45 skill pairs** described so similarly the model
|
|
108
|
+
can't reliably pick between them — so it fires the **wrong** skill. Valid markdown;
|
|
109
|
+
the selector chooses by description, and near-identical text collides.
|
|
110
|
+
**[How triggering works →](docs/measuring-skills.md)**
|
|
134
111
|
|
|
135
|
-
|
|
112
|
+
## Proof 2 — a tool your subagent silently can't call
|
|
136
113
|
|
|
137
|
-
```
|
|
138
|
-
|
|
114
|
+
```text
|
|
115
|
+
✗ tester — Tool "AskUserQuestion" is never available to a subagent.
|
|
116
|
+
→ remove or correct it — it's silently dropped from the contract.
|
|
139
117
|
```
|
|
140
118
|
|
|
141
|
-
|
|
142
|
-
|
|
143
|
-
**
|
|
144
|
-
|
|
145
|
-
|
|
146
|
-
|
|
147
|
-
- _"test my skills"_ → scaffolds **and runs** a trigger/behaviour test (`test-harness`)
|
|
148
|
-
- _"harden my rules"_ → upgrades prose guidance into enforced linter rules (`strengthen`)
|
|
149
|
-
- _"add a rule to my CLAUDE.md"_ → edits the source and recompiles (`edit-spec`)
|
|
150
|
-
|
|
151
|
-
<details>
|
|
152
|
-
<summary>What <code>init</code> sets up</summary>
|
|
153
|
-
|
|
154
|
-
- **Both lint and test** by default; scope with `--lint` / `--test`.
|
|
155
|
-
- **Already have a CLAUDE.md / AGENTS.md? `init` adopts it** into a spec faithfully and **non-destructively** — your file is left untouched until you choose to `compile` (and `eject` undoes it).
|
|
156
|
-
- Adds `vigiles` to `devDependencies`; installs the Claude Code plugin (skills + hooks) via the marketplace — globally, never vendored.
|
|
157
|
-
- Wires CI as a `zernie/vigiles@v1` workflow (a composite over the same CLI) that posts a sticky PR comment + a `valid` output.
|
|
158
|
-
|
|
159
|
-
Prefer to write tests yourself? They can be JS **or** TS
|
|
160
|
-
(`*.harness.{mjs,ts}`) — run them with `npx vigiles test`.
|
|
161
|
-
|
|
162
|
-
</details>
|
|
163
|
-
|
|
164
|
-
## ① Lint — your CLAUDE.md lies to your agent
|
|
119
|
+
A real upstream subagent declares a tool the harness **silently drops**, so it loses
|
|
120
|
+
a capability it thinks it has. vigiles flags it _and_ hands you the one-line fix —
|
|
121
|
+
**free, no model.** That's the difference from a markdown linter: it checks your
|
|
122
|
+
harness against **reality**, not style — every file path, script, code symbol, and
|
|
123
|
+
linter rule across **7 catalogs** (ESLint, Ruff, Clippy + four more), each verified
|
|
124
|
+
to **exist _and_ be enabled**. **[Full guide →](docs/verifying-instruction-files.md)**
|
|
165
125
|
|
|
166
|
-
|
|
167
|
-
|
|
168
|
-
ago and the script was renamed. The agent trusts the stale claim and acts on
|
|
169
|
-
fiction. `npx vigiles lint` resolves every reference against reality:
|
|
126
|
+
Two real catches, both **free and model-less** — and audit ranks a whole
|
|
127
|
+
marketplace the same way. **[Audit a marketplace →](docs/for-plugin-authors.md)**
|
|
170
128
|
|
|
171
|
-
|
|
172
|
-
|
|
173
|
-
✗ src/auth/login.ts — no such file (renamed or moved?)
|
|
174
|
-
✗ npm run check — not in package.json. Did you mean: "check:types"?
|
|
175
|
-
✓ @typescript-eslint/no-floating-promises — exists and enabled in eslint config
|
|
176
|
-
```
|
|
177
|
-
|
|
178
|
-
<p align="center">
|
|
179
|
-
<img src="vigiles-demo.gif" width="720" alt="vigiles lint catching a file that moved and a script that was renamed" />
|
|
180
|
-
</p>
|
|
181
|
-
<!-- Regenerate the GIF: `python3 scripts/make-demo-gif.py` (output is verbatim CLI; see scripts/demo.sh for a live asciinema recording). -->
|
|
129
|
+
> **And it grades itself: 100/100, A, all four rings green** — and CI gates every
|
|
130
|
+
> commit with `lint` + `test`. We eat what we cook.
|
|
182
131
|
|
|
183
|
-
|
|
184
|
-
(ESLint, Ruff, Clippy + four more): each rule exists **and is enabled**.
|
|
132
|
+
## How it does it — three instruments
|
|
185
133
|
|
|
186
|
-
|
|
187
|
-
|
|
188
|
-
inline `<!-- vigiles:enforce -->` comment lints too — no spec, no TypeScript. And
|
|
189
|
-
`vigiles eject` hands a spec back to markdown anytime.
|
|
190
|
-
**[Full guide →](docs/verifying-instruction-files.md)**
|
|
134
|
+
`Agent = Model + Harness`[^name]. The model gets the headlines; the **harness** is the
|
|
135
|
+
half you own. `audit` is the dashboard — these fix and prove what it finds.
|
|
191
136
|
|
|
192
|
-
|
|
193
|
-
> hand-off between agents becomes a build error instead of a runtime surprise —
|
|
194
|
-
> graduated like TypeScript's `strict`, on only when you want it.
|
|
195
|
-
> **[How →](docs/compiled-hooks.md)**
|
|
137
|
+
### 🔎 Lint — your CLAUDE.md stops lying
|
|
196
138
|
|
|
197
|
-
|
|
139
|
+
Every path, script, symbol & linter rule resolved against reality (the catches
|
|
140
|
+
above). **You don't write any of it** — `npx vigiles init` **adopts your existing
|
|
141
|
+
CLAUDE.md _and every skill and subagent_ into verified specs**, non-destructively
|
|
142
|
+
(untouched until you `compile`; `eject` reverses). After that, plain-English asks
|
|
143
|
+
edit them for you. **[How →](docs/verifying-instruction-files.md)**
|
|
198
144
|
|
|
199
|
-
|
|
200
|
-
to trigger, or hijack unrelated prompts; injected context can silently never reach
|
|
201
|
-
the model. All of it passes a naive "did it run?" check. vigiles tests the
|
|
202
|
-
assembled harness for real.
|
|
145
|
+
### 🧪 Test — does the harness actually do its job?
|
|
203
146
|
|
|
204
|
-
|
|
147
|
+
A hook that blocks nothing, a skill that hijacks unrelated prompts, injected context
|
|
148
|
+
that never reaches the model — all pass a naive "did it run?" check. Start at the
|
|
149
|
+
cheapest tier: a hook, called directly. **No model, no key:**
|
|
205
150
|
|
|
206
151
|
```typescript
|
|
207
152
|
import { runHook } from "vigiles/testing";
|
|
@@ -214,102 +159,79 @@ const r = runHook(guard, {
|
|
|
214
159
|
assert(r.blocked); // a red ✗ means your guard silently lets it through
|
|
215
160
|
```
|
|
216
161
|
|
|
217
|
-
|
|
162
|
+
Hooks **block** · skills **trigger** (recall _and_ precision) · subagents **finish**
|
|
163
|
+
in the outcome they promised (no LLM judge) · **safety holds** — `interceptTools`
|
|
164
|
+
catches a push or paid-API call as an _attempt_, so the side effect never happens.
|
|
165
|
+
Almost every tier runs with **no model and no key**, on every commit.
|
|
166
|
+
**[How testing works →](docs/harness-testing.md)**
|
|
218
167
|
|
|
219
|
-
|
|
220
|
-
- **Skills trigger** on the right prompts and stay quiet on the wrong ones — recall _and_ precision (`measureTriggerRate`).
|
|
221
|
-
- **Subagents finish right** — assert a subagent ended in the success (or error) outcome it promised, with a plain check, no LLM judge (`assertAgentOk` / `assertAgentErr`).
|
|
222
|
-
- **Behaviour is good** — score a skill's output, or A/B it on-vs-off for the real lift (`measure` / `runEval`, with significance testing).
|
|
223
|
-
- **Safety holds** — the agent _didn't_ push to the wrong branch or hit a paid API; `interceptTools` catches the attempt so the side effect never happens.
|
|
168
|
+
### 📊 Eval — does a skill help, or just cost more?
|
|
224
169
|
|
|
225
|
-
|
|
226
|
-
|
|
227
|
-
|
|
170
|
+
_"65% fewer tokens." Says who?_ vigiles A/Bs the claim on real coding tasks and reports
|
|
171
|
+
the **bill**, the **target it claims to move**, and the **blast radius** (did the code
|
|
172
|
+
still work?).
|
|
228
173
|
|
|
229
|
-
|
|
174
|
+
**The eval you can actually afford:** promptfoo / DeepEval hit a metered API and bill
|
|
175
|
+
**per token, every run**. vigiles answers most questions with **no model at all**, and
|
|
176
|
+
runs the rest on your own **Claude Pro/Max subscription — $0 extra.**
|
|
177
|
+
**[Measure a skill →](docs/measuring-skills.md)**
|
|
230
178
|
|
|
231
|
-
|
|
232
|
-
it — stars and vibes, **zero measurement**. vigiles A/Bs the claim on real coding
|
|
233
|
-
tasks, the harness loaded exactly as it ships, and reports **three numbers**:
|
|
179
|
+
## Quick start
|
|
234
180
|
|
|
235
|
-
|
|
236
|
-
import { measureArms } from "vigiles/testing";
|
|
181
|
+
**Paste into Claude Code or Codex:**
|
|
237
182
|
|
|
238
|
-
|
|
239
|
-
|
|
240
|
-
|
|
241
|
-
|
|
242
|
-
|
|
243
|
-
|
|
183
|
+
```text
|
|
184
|
+
Set up vigiles in this repo: run `npx vigiles init` and accept the defaults. If I
|
|
185
|
+
already have a CLAUDE.md or AGENTS.md, adopt it into a spec and show me which
|
|
186
|
+
references are stale. Then install the dep, compile, and write + run one harness
|
|
187
|
+
test for a hook or skill of mine. Don't enforce a spec-per-file or add a real-model
|
|
188
|
+
eval without asking me first.
|
|
244
189
|
```
|
|
245
190
|
|
|
246
|
-
|
|
247
|
-
- **The target** — whatever the skill claims to move (output tokens, latency, tool calls), verified on its own terms.
|
|
248
|
-
- **The blast radius** — correctness, a deterministic 1/0. A token win that breaks the code is **not a win**.
|
|
249
|
-
|
|
250
|
-
**Safe to repeat.** Each real-model run is sandboxed (ephemeral dir, egress blocked
|
|
251
|
-
or allow-listed), and `interceptTools` catches an irreversible external — a push, a
|
|
252
|
-
paid API — as an _attempt_, never running it. **[Safety, sandboxing & FAQ →](docs/safety.md)**
|
|
253
|
-
|
|
254
|
-
**The eval you can actually afford.** promptfoo / DeepEval hit a metered API and
|
|
255
|
-
bill **per token, every run**. vigiles answers most questions with **no model at
|
|
256
|
-
all**, and runs the rest on your own **Claude Pro/Max subscription — $0 extra**. So
|
|
257
|
-
you can measure on every change. **[Eval a skill →](docs/measuring-skills.md)** · **[Why it's affordable →](docs/eval-architecture.md)**
|
|
191
|
+
Or do it yourself:
|
|
258
192
|
|
|
259
|
-
|
|
193
|
+
```bash
|
|
194
|
+
npx vigiles init # lint + test: spec + harness test + CI + plugin
|
|
195
|
+
```
|
|
260
196
|
|
|
261
|
-
|
|
197
|
+
Interactive in a terminal, non-interactive for agents/CI (or `--yes`).
|
|
262
198
|
|
|
263
|
-
**
|
|
264
|
-
|
|
265
|
-
you. _(Already write safety hooks? This is the power tool.)_ Write a pure typed
|
|
266
|
-
function instead; vigiles emits the exit code, the JSON, and an AST-backed matcher:
|
|
199
|
+
**You don't hand-write any of this — your agent does.** `init` installs
|
|
200
|
+
model-invocable skills, so a plain-English ask does the work:
|
|
267
201
|
|
|
268
|
-
|
|
269
|
-
|
|
270
|
-
|
|
271
|
-
export default defineHook({
|
|
272
|
-
on: "PreToolUse",
|
|
273
|
-
match: tool("Bash"),
|
|
274
|
-
decide: (e) =>
|
|
275
|
-
e.command.runs("git push", { force: true })
|
|
276
|
-
? deny("no force-push to a protected branch")
|
|
277
|
-
: allow(),
|
|
278
|
-
});
|
|
279
|
-
```
|
|
202
|
+
- _"test my skills"_ → scaffolds **and runs** a trigger/behaviour test (`test-harness`)
|
|
203
|
+
- _"harden my rules"_ → upgrades prose guidance into enforced linter rules (`strengthen`)
|
|
204
|
+
- _"add a rule to my CLAUDE.md"_ → edits the source and recompiles (`edit-spec`)
|
|
280
205
|
|
|
281
|
-
|
|
282
|
-
|
|
206
|
+
<details>
|
|
207
|
+
<summary>What <code>init</code> sets up</summary>
|
|
283
208
|
|
|
284
|
-
|
|
285
|
-
|
|
286
|
-
the
|
|
209
|
+
- **Both lint and test** by default; scope with `--lint` / `--test`.
|
|
210
|
+
- **Already have a CLAUDE.md / AGENTS.md, skills, or subagents? `init` adopts them all** into specs faithfully and **non-destructively** — untouched until you `compile` (and `eject` undoes it).
|
|
211
|
+
- Adds `vigiles` to `devDependencies`; installs the Claude Code plugin (skills + hooks) via the marketplace — globally, never vendored.
|
|
212
|
+
- Wires CI as a `zernie/vigiles@v1` workflow that posts a sticky PR comment + a `valid` output.
|
|
287
213
|
|
|
288
|
-
|
|
289
|
-
|
|
290
|
-
(
|
|
291
|
-
default, not an unbypassable wall._
|
|
292
|
-
**[Compiled hooks — bug classes + trade-offs →](docs/compiled-hooks.md)**
|
|
214
|
+
Works with **Claude Code and Codex** ([`vigiles/codex`](docs/harnesses.md)) or
|
|
215
|
+
[your own harness](docs/authoring-an-adapter.md). Prefer to write tests yourself?
|
|
216
|
+
JS **or** TS (`*.harness.{mjs,ts}`) — run with `npx vigiles test`.
|
|
293
217
|
|
|
294
|
-
|
|
218
|
+
</details>
|
|
295
219
|
|
|
296
220
|
## FAQ
|
|
297
221
|
|
|
298
222
|
- **Isn't this just a markdown linter?** No — it checks whether your instruction file is _true_ (every path/script/symbol/rule exists and is enabled), then tests and measures your harness. A style linter can't do any of that.
|
|
299
|
-
- **Do I have to write TypeScript?** No — your agent writes the spec (`init` adopts your CLAUDE.md into one)
|
|
300
|
-
- **Does it overwrite my files?** No. `init` adopts an existing CLAUDE.md _non-destructively_ — untouched until you `compile`, and `eject` reverses it.
|
|
301
|
-
- **Need an API key?** No for almost everything (free, every commit). Real-model evals run on your Claude Pro/Max subscription — $0 metered tokens.
|
|
223
|
+
- **Do I have to write TypeScript?** No — your agent writes the spec (`init` adopts your CLAUDE.md into one), or plain markdown lints with zero new files. Compiler-grade guarantees are opt-in, like TS's `strict`.
|
|
302
224
|
- **Non-JS repo?** `npx vigiles lint` verifies your CLAUDE.md with no install (Ruff/Clippy/Pylint/… too).
|
|
303
225
|
|
|
304
226
|
**[Full FAQ →](docs/faq.md)**
|
|
305
227
|
|
|
306
228
|
## More
|
|
307
229
|
|
|
308
|
-
- **[CLI →](docs/cli.md)**
|
|
230
|
+
- **[CLI →](docs/cli.md)** · **[GitHub Action →](docs/github-action.md)** · the full **[lint rules matrix →](docs/verifying-instruction-files.md#the-validation-rules--the-full-matrix)** lives with the linting guide.
|
|
309
231
|
- **[Skills →](docs/skills.md)** — the skills `init` installs, and how the model-invocable ones trigger.
|
|
310
|
-
- **[Ship plugins? The plugin-author guide →](docs/for-plugin-authors.md)** — scan a draft
|
|
311
|
-
- **[Docs index →](docs/README.md)** · **[API reference →](https://zernie.github.io/vigiles/)** · **[Related tools →](docs/related-tools.md)
|
|
312
|
-
- **[Stability →](STABILITY.md)** — 0.x: the CLI is stable; the library API is still evolving
|
|
232
|
+
- **[Ship plugins? The plugin-author guide →](docs/for-plugin-authors.md)** — scan a draft, make your skills fire, rank a whole marketplace — no key.
|
|
233
|
+
- **[Docs index →](docs/README.md)** · **[API reference →](https://zernie.github.io/vigiles/)** · **[Related tools →](docs/related-tools.md)**.
|
|
234
|
+
- **[Stability →](STABILITY.md)** — 0.x: the CLI is stable; the library API is still evolving.
|
|
313
235
|
- **Not for you if** you want a model/capability benchmark or runtime guardrails in the request path — vigiles is build-/CI-time.
|
|
314
236
|
- Companion to [Feedback Loop Is All You Need](https://zernie.com/blog/feedback-loop-is-all-you-need).
|
|
315
237
|
|
package/dist/audit-html.d.ts
CHANGED
|
@@ -5,16 +5,27 @@ import type { AuditReport } from "./audit-report.js";
|
|
|
5
5
|
* we use `__dirname`, not `import.meta`.
|
|
6
6
|
*/
|
|
7
7
|
export declare function templatePath(): string | null;
|
|
8
|
+
/**
|
|
9
|
+
* Live-server info injected into the report when it's served by `audit --serve`
|
|
10
|
+
* (not a static file): the per-run token the report must send on every adopt
|
|
11
|
+
* POST. Absent for a plain static report — the buttons fall back to copy-command.
|
|
12
|
+
*/
|
|
13
|
+
export interface ServeInfo {
|
|
14
|
+
readonly token: string;
|
|
15
|
+
}
|
|
8
16
|
/**
|
|
9
17
|
* Inject the report JSON into a template by replacing the quoted placeholder
|
|
10
18
|
* string with the JSON object literal. Pure — the testable core. Throws if the
|
|
11
|
-
* template is missing the placeholder.
|
|
19
|
+
* template is missing the placeholder. When `serve` is given (the `--serve`
|
|
20
|
+
* path), also sets `window.__VIGILES_SERVE__` so the report's adopt buttons POST
|
|
21
|
+
* to the local server instead of copying the command.
|
|
12
22
|
*/
|
|
13
|
-
export declare function injectReportData(template: string, report: AuditReport): string;
|
|
23
|
+
export declare function injectReportData(template: string, report: AuditReport, serve?: ServeInfo): string;
|
|
14
24
|
/**
|
|
15
25
|
* Render the self-contained HTML report (React template + injected data). Throws
|
|
16
26
|
* if the template hasn't been built — the caller (writeAuditHtml) catches that and
|
|
17
|
-
* skips the HTML, since the JSON + terminal report don't depend on it.
|
|
27
|
+
* skips the HTML, since the JSON + terminal report don't depend on it. Pass
|
|
28
|
+
* `serve` to render the live (`--serve`) variant whose buttons POST.
|
|
18
29
|
*/
|
|
19
|
-
export declare function renderAuditHtml(report: AuditReport): string;
|
|
30
|
+
export declare function renderAuditHtml(report: AuditReport, serve?: ServeInfo): string;
|
|
20
31
|
//# sourceMappingURL=audit-html.d.ts.map
|
package/dist/audit-html.js
CHANGED
|
@@ -37,25 +37,34 @@ function escapeForScript(json) {
|
|
|
37
37
|
/**
|
|
38
38
|
* Inject the report JSON into a template by replacing the quoted placeholder
|
|
39
39
|
* string with the JSON object literal. Pure — the testable core. Throws if the
|
|
40
|
-
* template is missing the placeholder.
|
|
40
|
+
* template is missing the placeholder. When `serve` is given (the `--serve`
|
|
41
|
+
* path), also sets `window.__VIGILES_SERVE__` so the report's adopt buttons POST
|
|
42
|
+
* to the local server instead of copying the command.
|
|
41
43
|
*/
|
|
42
|
-
function injectReportData(template, report) {
|
|
44
|
+
function injectReportData(template, report, serve) {
|
|
43
45
|
const re = new RegExp(`(["'])${PLACEHOLDER}\\1`);
|
|
44
46
|
if (!re.test(template)) {
|
|
45
47
|
throw new Error("audit report template is missing the data placeholder");
|
|
46
48
|
}
|
|
47
|
-
|
|
49
|
+
let out = template.replace(re, escapeForScript(JSON.stringify(report)));
|
|
50
|
+
if (serve) {
|
|
51
|
+
// Prepend the serve global onto the same statement that carries the data, so
|
|
52
|
+
// it's set before the app bundle reads it. The token is hex, but escape anyway.
|
|
53
|
+
out = out.replace("window.__VIGILES_DATA__", `window.__VIGILES_SERVE__=${escapeForScript(JSON.stringify(serve))};window.__VIGILES_DATA__`);
|
|
54
|
+
}
|
|
55
|
+
return out;
|
|
48
56
|
}
|
|
49
57
|
/**
|
|
50
58
|
* Render the self-contained HTML report (React template + injected data). Throws
|
|
51
59
|
* if the template hasn't been built — the caller (writeAuditHtml) catches that and
|
|
52
|
-
* skips the HTML, since the JSON + terminal report don't depend on it.
|
|
60
|
+
* skips the HTML, since the JSON + terminal report don't depend on it. Pass
|
|
61
|
+
* `serve` to render the live (`--serve`) variant whose buttons POST.
|
|
53
62
|
*/
|
|
54
|
-
function renderAuditHtml(report) {
|
|
63
|
+
function renderAuditHtml(report, serve) {
|
|
55
64
|
const p = templatePath();
|
|
56
65
|
if (!p) {
|
|
57
66
|
throw new Error("audit report template not built — run `npm run build` (builds report/), or use --json / --no-html");
|
|
58
67
|
}
|
|
59
|
-
return injectReportData((0, node_fs_1.readFileSync)(p, "utf-8"), report);
|
|
68
|
+
return injectReportData((0, node_fs_1.readFileSync)(p, "utf-8"), report, serve);
|
|
60
69
|
}
|
|
61
70
|
//# sourceMappingURL=audit-html.js.map
|