vigiles 2.4.0 → 2.5.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +54 -3
- package/dist/agent-result.d.ts +40 -0
- package/dist/agent-result.js +97 -0
- package/dist/agent-runtime.d.ts +64 -0
- package/dist/agent-runtime.js +147 -0
- package/dist/cli.js +106 -0
- package/dist/compile.d.ts +32 -3
- package/dist/compile.js +268 -0
- package/dist/eval-cache.d.ts +33 -0
- package/dist/eval-cache.js +94 -0
- package/dist/eval.d.ts +172 -9
- package/dist/eval.js +319 -58
- package/dist/harness-assert.d.ts +140 -12
- package/dist/harness-assert.js +283 -25
- package/dist/harness-test.d.ts +97 -11
- package/dist/harness-test.js +147 -37
- package/dist/judge.js +2 -0
- package/dist/linters.d.ts +6 -0
- package/dist/linters.js +1 -0
- package/dist/mock-entry.d.ts +2 -0
- package/dist/mock-entry.js +36 -0
- package/dist/mock-model.d.ts +29 -0
- package/dist/mock-model.js +40 -0
- package/dist/plugin-loader.js +51 -17
- package/dist/sandbox.d.ts +76 -0
- package/dist/sandbox.js +241 -0
- package/dist/spec.d.ts +130 -0
- package/dist/spec.js +55 -0
- package/dist/stats.d.ts +49 -0
- package/dist/stats.js +109 -0
- package/package.json +5 -3
package/README.md
CHANGED
|
@@ -313,7 +313,8 @@ assert(JSON.parse(r.stdout).num_turns > 1); // the Stop hook forced more work
|
|
|
313
313
|
### Level 3 — does it change what Claude does? (real AI, occasional)
|
|
314
314
|
|
|
315
315
|
`runEval` runs the **real** model N times with your change **on vs off** and
|
|
316
|
-
reports the gap
|
|
316
|
+
reports the gap as **mean ± se** — so you can tell signal from noise instead of
|
|
317
|
+
eyeballing two averages:
|
|
317
318
|
|
|
318
319
|
```typescript
|
|
319
320
|
import { runEval, formatEvalReport } from "vigiles/eval";
|
|
@@ -325,10 +326,21 @@ const report = await runEval({
|
|
|
325
326
|
marked: ctx.sh("grep -c vigiles:symbol SKILL.md") !== "0",
|
|
326
327
|
}),
|
|
327
328
|
trials: 6,
|
|
329
|
+
cache: "readwrite", // replay past runs — editing `measure` re-scores for free
|
|
328
330
|
});
|
|
329
|
-
console.log(formatEvalReport(report));
|
|
331
|
+
console.log(formatEvalReport(report));
|
|
332
|
+
// off marked=0.00 on marked=0.50±0.20 pass^k=0 ($0.07 · 1.2s/run · 4.1k tok)
|
|
330
333
|
```
|
|
331
334
|
|
|
335
|
+
`assertSignificant(report, { baseline: "off", arm: "on", metric: "marked" })`
|
|
336
|
+
turns the gap into a CI gate — a Welch t-test decides whether it cleared the
|
|
337
|
+
noise floor, **computed** from the arms' spread, not hand-fed. Runs go
|
|
338
|
+
**concurrently**, track **cost / latency / tokens** (cap them with `maxCostUsd`),
|
|
339
|
+
and the **record/replay cache** makes re-scoring after a `measure` edit free.
|
|
340
|
+
|
|
341
|
+
Same tier, different question: **`measureTriggerRate`** measures how reliably a
|
|
342
|
+
skill's _description fires_ across varied prompts — the #1 skill-authoring pain.
|
|
343
|
+
|
|
332
344
|
### Test your skills for real — and assert on what Claude _did_
|
|
333
345
|
|
|
334
346
|
Install a plugin the way Claude actually does (`pluginDir` → `--plugin-dir`) so
|
|
@@ -356,6 +368,43 @@ dangerous tool was never used, which "the file looks unchanged" can't. It works
|
|
|
356
368
|
on **real third-party plugins** too: the suite confirms real `obra/superpowers`
|
|
357
369
|
and `wshobson/agents` skills resolve this way, with no markers injected.
|
|
358
370
|
|
|
371
|
+
### Did the injected context actually reach the model?
|
|
372
|
+
|
|
373
|
+
A SessionStart hook or a slash command can _fire_ and still inject **nothing** —
|
|
374
|
+
wrong output shape, wrong platform. `trace.modelRequests` records what the model
|
|
375
|
+
actually received (system + messages), so you assert it landed, not just that the
|
|
376
|
+
hook ran — **"fired ≠ landed"**:
|
|
377
|
+
|
|
378
|
+
```typescript
|
|
379
|
+
import { assertRequestContains } from "vigiles/harness-assert";
|
|
380
|
+
|
|
381
|
+
assertRequestContains(r, "You have superpowers"); // the additionalContext reached the model
|
|
382
|
+
```
|
|
383
|
+
|
|
384
|
+
(Dogfood: this is exactly how vigiles found that real `obra/superpowers` emits a
|
|
385
|
+
_top-level_ `additionalContext`, which Claude Code — reading the _nested_ form —
|
|
386
|
+
never injects. The hook fired; the context never landed.)
|
|
387
|
+
|
|
388
|
+
### Running an untrusted plugin? It's confined by default
|
|
389
|
+
|
|
390
|
+
Testing a third-party plugin means executing **its** hooks. `runHarnessTest` is
|
|
391
|
+
safe by default: code you wrote (inline `settings`/`files`) runs directly, but an
|
|
392
|
+
external `plugin` / `pluginDir` is **confined under bubblewrap** — a network
|
|
393
|
+
namespace with **no egress** (a malicious hook can't phone home), a read-only
|
|
394
|
+
filesystem, and a **cleared environment** (your `ANTHROPIC_API_KEY` and other
|
|
395
|
+
secrets aren't even visible). If no sandbox is available the run **refuses**
|
|
396
|
+
rather than executing unconfined:
|
|
397
|
+
|
|
398
|
+
```typescript
|
|
399
|
+
runHarnessTest({ pluginDir: "./vendor/some-plugin", model }); // confined, or refuses
|
|
400
|
+
runHarnessTest({ pluginDir: "./audited", model, sandbox: false }); // you vouch for it → direct
|
|
401
|
+
```
|
|
402
|
+
|
|
403
|
+
Confinement is **Linux-only** (bubblewrap); on macOS / Windows an untrusted run
|
|
404
|
+
refuses unless you pass `sandbox: false`. The suite dogfoods it on real
|
|
405
|
+
`obra/superpowers` — its `SessionStart` hook runs in a no-egress sandbox, and the
|
|
406
|
+
test proves egress is blocked while the scripted mock stays reachable.
|
|
407
|
+
|
|
359
408
|
### Run them in CI
|
|
360
409
|
|
|
361
410
|
`vigiles test` runs `*.harness.mjs` files (free, no key); `vigiles eval` runs
|
|
@@ -379,10 +428,12 @@ npx vigiles eval --trials=6 examples/harness/skill-outcome.eval.mjs
|
|
|
379
428
|
| Hooks — PreCompact / Notification / SessionEnd / SubagentStop | ✅ logic | — (mock can't trigger) | 🟡 |
|
|
380
429
|
| CLAUDE.md / instructions | ✅ refs | 🟡 present, not behaviour | ✅ behaviour |
|
|
381
430
|
| Skills | 🟡 refs | ✅ resolves via `pluginDir` | ✅ activation |
|
|
382
|
-
| Subagents (`agents/`) | 🟡 refs
|
|
431
|
+
| Subagents (`agents/`) | ✅ tool rail · 🟡 refs | 🟡 rail not live-armed | ✅ via Task |
|
|
383
432
|
| Slash commands (`commands/`) | 🟡 refs | 🟡 needs prompt capture | ✅ via `/cmd` |
|
|
384
433
|
| MCP servers | ✅ tool refs (`vigiles:mcp`) | 🔴 | 🔴 |
|
|
385
434
|
| settings.json | 🟡 assert merged | ✅ applied | ✅ |
|
|
435
|
+
| Hook context injection (does it _land_?) | — n/a | ✅ `trace.modelRequests` | ✅ |
|
|
436
|
+
| Untrusted plugin execution | — n/a | ✅ confined (bwrap, Linux) | 🟡 outer sandbox |
|
|
386
437
|
|
|
387
438
|
✅ shipped · 🟡 partial · 🔴 gap · — n/a. Full detail + roadmap: [`research/harness-testing-coverage-matrix.md`](research/harness-testing-coverage-matrix.md).
|
|
388
439
|
|
|
@@ -0,0 +1,40 @@
|
|
|
1
|
+
/**
|
|
2
|
+
* vigiles — parse a subagent's railway result.
|
|
3
|
+
*
|
|
4
|
+
* A subagent with a `result()` contract is told (in its compiled system prompt)
|
|
5
|
+
* to end its turn with exactly one fenced block:
|
|
6
|
+
*
|
|
7
|
+
* ```vigiles:ok
|
|
8
|
+
* { "files": ["a.ts"], "summary": "done" }
|
|
9
|
+
* ```
|
|
10
|
+
*
|
|
11
|
+
* or `vigiles:err` for the error track. This module extracts and validates that
|
|
12
|
+
* block — the single primitive the railway orchestrator and the harness-test
|
|
13
|
+
* assertions (`assertAgentOk`/`assertAgentErr`) both build on. Pure and
|
|
14
|
+
* model-free: hand it the worker's text, get back a discriminated outcome.
|
|
15
|
+
*
|
|
16
|
+
* "Railway-oriented" is literal here: the parse is `text -> Result<S, E>` with a
|
|
17
|
+
* third `malformed` track for a worker that didn't honor its contract (no block,
|
|
18
|
+
* bad JSON, or a shape that doesn't match the declared schema).
|
|
19
|
+
*/
|
|
20
|
+
import type { OutputContract } from "./spec.js";
|
|
21
|
+
/** The outcome of parsing a worker's result block. */
|
|
22
|
+
export type ParsedAgentResult<S = Record<string, unknown>, E = Record<string, unknown>> = {
|
|
23
|
+
readonly kind: "ok";
|
|
24
|
+
readonly value: S;
|
|
25
|
+
} | {
|
|
26
|
+
readonly kind: "err";
|
|
27
|
+
readonly error: E;
|
|
28
|
+
} | {
|
|
29
|
+
readonly kind: "malformed";
|
|
30
|
+
readonly reason: string;
|
|
31
|
+
};
|
|
32
|
+
/**
|
|
33
|
+
* Parse the last `vigiles:ok` / `vigiles:err` block from a worker's output.
|
|
34
|
+
*
|
|
35
|
+
* With a `contract`, the parsed object is validated against the matching track's
|
|
36
|
+
* shape — a worker that emits the wrong shape is `malformed`, not a silent pass.
|
|
37
|
+
* Without one, any well-formed JSON block is accepted.
|
|
38
|
+
*/
|
|
39
|
+
export declare function parseAgentResult(text: string, contract?: OutputContract): ParsedAgentResult;
|
|
40
|
+
//# sourceMappingURL=agent-result.d.ts.map
|
|
@@ -0,0 +1,97 @@
|
|
|
1
|
+
"use strict";
|
|
2
|
+
/**
|
|
3
|
+
* vigiles — parse a subagent's railway result.
|
|
4
|
+
*
|
|
5
|
+
* A subagent with a `result()` contract is told (in its compiled system prompt)
|
|
6
|
+
* to end its turn with exactly one fenced block:
|
|
7
|
+
*
|
|
8
|
+
* ```vigiles:ok
|
|
9
|
+
* { "files": ["a.ts"], "summary": "done" }
|
|
10
|
+
* ```
|
|
11
|
+
*
|
|
12
|
+
* or `vigiles:err` for the error track. This module extracts and validates that
|
|
13
|
+
* block — the single primitive the railway orchestrator and the harness-test
|
|
14
|
+
* assertions (`assertAgentOk`/`assertAgentErr`) both build on. Pure and
|
|
15
|
+
* model-free: hand it the worker's text, get back a discriminated outcome.
|
|
16
|
+
*
|
|
17
|
+
* "Railway-oriented" is literal here: the parse is `text -> Result<S, E>` with a
|
|
18
|
+
* third `malformed` track for a worker that didn't honor its contract (no block,
|
|
19
|
+
* bad JSON, or a shape that doesn't match the declared schema).
|
|
20
|
+
*/
|
|
21
|
+
Object.defineProperty(exports, "__esModule", { value: true });
|
|
22
|
+
exports.parseAgentResult = parseAgentResult;
|
|
23
|
+
// Capture every vigiles:ok / vigiles:err fenced block; the LAST one is the
|
|
24
|
+
// worker's final answer (earlier ones may be illustrative in its reasoning).
|
|
25
|
+
const BLOCK_RE = /```vigiles:(ok|err)[ \t]*\r?\n([\s\S]*?)```/g;
|
|
26
|
+
/** Does a runtime value match a declared field type? */
|
|
27
|
+
function fieldMatches(value, type) {
|
|
28
|
+
switch (type) {
|
|
29
|
+
case "string":
|
|
30
|
+
return typeof value === "string";
|
|
31
|
+
case "number":
|
|
32
|
+
return typeof value === "number";
|
|
33
|
+
case "boolean":
|
|
34
|
+
return typeof value === "boolean";
|
|
35
|
+
case "string[]":
|
|
36
|
+
return Array.isArray(value) && value.every((v) => typeof v === "string");
|
|
37
|
+
}
|
|
38
|
+
}
|
|
39
|
+
/** Validate a parsed object against a contract track; null when it conforms. */
|
|
40
|
+
function shapeError(obj, shape) {
|
|
41
|
+
for (const [field, type] of Object.entries(shape)) {
|
|
42
|
+
if (!(field in obj))
|
|
43
|
+
return `missing field "${field}"`;
|
|
44
|
+
if (!fieldMatches(obj[field], type)) {
|
|
45
|
+
return `field "${field}" should be ${type}`;
|
|
46
|
+
}
|
|
47
|
+
}
|
|
48
|
+
return null;
|
|
49
|
+
}
|
|
50
|
+
/**
|
|
51
|
+
* Parse the last `vigiles:ok` / `vigiles:err` block from a worker's output.
|
|
52
|
+
*
|
|
53
|
+
* With a `contract`, the parsed object is validated against the matching track's
|
|
54
|
+
* shape — a worker that emits the wrong shape is `malformed`, not a silent pass.
|
|
55
|
+
* Without one, any well-formed JSON block is accepted.
|
|
56
|
+
*/
|
|
57
|
+
function parseAgentResult(text, contract) {
|
|
58
|
+
BLOCK_RE.lastIndex = 0;
|
|
59
|
+
let last = null;
|
|
60
|
+
for (let m = BLOCK_RE.exec(text); m !== null; m = BLOCK_RE.exec(text)) {
|
|
61
|
+
last = { track: m[1], body: m[2] };
|
|
62
|
+
}
|
|
63
|
+
if (!last) {
|
|
64
|
+
return {
|
|
65
|
+
kind: "malformed",
|
|
66
|
+
reason: "no vigiles:ok/vigiles:err block found",
|
|
67
|
+
};
|
|
68
|
+
}
|
|
69
|
+
let parsed;
|
|
70
|
+
try {
|
|
71
|
+
parsed = JSON.parse(last.body);
|
|
72
|
+
}
|
|
73
|
+
catch {
|
|
74
|
+
return {
|
|
75
|
+
kind: "malformed",
|
|
76
|
+
reason: `invalid JSON in vigiles:${last.track} block`,
|
|
77
|
+
};
|
|
78
|
+
}
|
|
79
|
+
if (typeof parsed !== "object" || parsed === null || Array.isArray(parsed)) {
|
|
80
|
+
return {
|
|
81
|
+
kind: "malformed",
|
|
82
|
+
reason: `vigiles:${last.track} block must be a JSON object`,
|
|
83
|
+
};
|
|
84
|
+
}
|
|
85
|
+
const obj = parsed;
|
|
86
|
+
if (contract) {
|
|
87
|
+
const shape = last.track === "ok" ? contract.ok : contract.err;
|
|
88
|
+
const err = shapeError(obj, shape);
|
|
89
|
+
if (err) {
|
|
90
|
+
return { kind: "malformed", reason: `${last.track} block: ${err}` };
|
|
91
|
+
}
|
|
92
|
+
}
|
|
93
|
+
return last.track === "ok"
|
|
94
|
+
? { kind: "ok", value: obj }
|
|
95
|
+
: { kind: "err", error: obj };
|
|
96
|
+
}
|
|
97
|
+
//# sourceMappingURL=agent-result.js.map
|
|
@@ -0,0 +1,64 @@
|
|
|
1
|
+
/**
|
|
2
|
+
* vigiles — Agent runtime: the PreToolUse tool-contract rail.
|
|
3
|
+
*
|
|
4
|
+
* A subagent declares an allowed-tools contract in its frontmatter (`tools:`).
|
|
5
|
+
* But that field is documentation, not a hard runtime boundary (Claude Code
|
|
6
|
+
* issue #54898): permissions are session-wide, a subagent inherits the parent
|
|
7
|
+
* session's grants, and `tools:` only filters what's *offered* — it can't deny
|
|
8
|
+
* what the session allows. The deterministic layer that actually closes the gap
|
|
9
|
+
* is a **PreToolUse hook** that blocks any tool the active agent's contract
|
|
10
|
+
* doesn't list.
|
|
11
|
+
*
|
|
12
|
+
* This is the same emit-a-hook pattern the skill runtime already ships
|
|
13
|
+
* (`src/skill-runtime.ts`): there a `Stop` hook reads the active skill's
|
|
14
|
+
* compiled SKILL.md and runs its result gate; here a `PreToolUse` hook reads
|
|
15
|
+
* the active agent's compiled `.md`, parses its `tools:` allowlist, and
|
|
16
|
+
* allows/denies the tool call. The compiled markdown's frontmatter is the
|
|
17
|
+
* single source of truth — the same list that documents intent IS the list the
|
|
18
|
+
* hook enforces, so the two agree by construction (see `enforcedTools`).
|
|
19
|
+
*
|
|
20
|
+
* Which agent is active is recorded in `.vigiles/active-agent.json` — Claude
|
|
21
|
+
* Code hooks don't surface the dispatched subagent, so vigiles records it
|
|
22
|
+
* (mirrors `.vigiles/active-skill.json`). The decision logic below is
|
|
23
|
+
* harness-agnostic and fully testable.
|
|
24
|
+
*/
|
|
25
|
+
/**
|
|
26
|
+
* Parse an agent's allowed-tools contract from its compiled markdown.
|
|
27
|
+
*
|
|
28
|
+
* Returns the list of allowed tool names, or `null` when the agent declares no
|
|
29
|
+
* `tools:` line at all — which in Claude Code means it inherits EVERY tool (the
|
|
30
|
+
* #1 footgun). `null` is the "no restriction" signal the decision logic honors;
|
|
31
|
+
* an empty list (`tools:` with nothing after it) means "no tools allowed".
|
|
32
|
+
*/
|
|
33
|
+
export declare function parseAgentTools(markdown: string): string[] | null;
|
|
34
|
+
export interface PreToolDecision {
|
|
35
|
+
/** Whether the tool call is allowed (true) or blocked (false). */
|
|
36
|
+
readonly allow: boolean;
|
|
37
|
+
/** Message fed back to the model on a block; empty on allow. */
|
|
38
|
+
readonly message: string;
|
|
39
|
+
}
|
|
40
|
+
/**
|
|
41
|
+
* Decide whether `tool` is allowed under an agent's tool contract. Pure, so the
|
|
42
|
+
* rail is unit-testable without spawning anything.
|
|
43
|
+
*
|
|
44
|
+
* - `allowed === null` → the agent declared no `tools:` line, so it inherits
|
|
45
|
+
* everything and the rail imposes no restriction (allow).
|
|
46
|
+
* - otherwise → allow iff the tool is in the allowlist; deny anything else,
|
|
47
|
+
* feeding the contract back to the model so it self-corrects.
|
|
48
|
+
*/
|
|
49
|
+
export declare function decidePreToolUse(allowed: readonly string[] | null, tool: string): PreToolDecision;
|
|
50
|
+
/** Record the subagent currently dispatched, so PreToolUse enforces its contract. */
|
|
51
|
+
export declare function setActiveAgent(cwd: string, agentPath: string): void;
|
|
52
|
+
/** Clear the active-agent marker (the subagent finished). */
|
|
53
|
+
export declare function clearActiveAgent(cwd: string): void;
|
|
54
|
+
/** The path of the active agent's compiled `.md`, or null when none is active. */
|
|
55
|
+
export declare function readActiveAgent(cwd: string): string | null;
|
|
56
|
+
/**
|
|
57
|
+
* PreToolUse-hook decision. If an agent is active, parse its compiled `.md`
|
|
58
|
+
* tool contract and allow the call only when the tool is in the allowlist;
|
|
59
|
+
* otherwise block and tell the model which tools it may use. With no active
|
|
60
|
+
* agent (or an agent that inherits all tools), always allow — the rail only
|
|
61
|
+
* constrains agents that declared a contract.
|
|
62
|
+
*/
|
|
63
|
+
export declare function evaluatePreToolUse(cwd: string, tool: string): PreToolDecision;
|
|
64
|
+
//# sourceMappingURL=agent-runtime.d.ts.map
|
|
@@ -0,0 +1,147 @@
|
|
|
1
|
+
"use strict";
|
|
2
|
+
/**
|
|
3
|
+
* vigiles — Agent runtime: the PreToolUse tool-contract rail.
|
|
4
|
+
*
|
|
5
|
+
* A subagent declares an allowed-tools contract in its frontmatter (`tools:`).
|
|
6
|
+
* But that field is documentation, not a hard runtime boundary (Claude Code
|
|
7
|
+
* issue #54898): permissions are session-wide, a subagent inherits the parent
|
|
8
|
+
* session's grants, and `tools:` only filters what's *offered* — it can't deny
|
|
9
|
+
* what the session allows. The deterministic layer that actually closes the gap
|
|
10
|
+
* is a **PreToolUse hook** that blocks any tool the active agent's contract
|
|
11
|
+
* doesn't list.
|
|
12
|
+
*
|
|
13
|
+
* This is the same emit-a-hook pattern the skill runtime already ships
|
|
14
|
+
* (`src/skill-runtime.ts`): there a `Stop` hook reads the active skill's
|
|
15
|
+
* compiled SKILL.md and runs its result gate; here a `PreToolUse` hook reads
|
|
16
|
+
* the active agent's compiled `.md`, parses its `tools:` allowlist, and
|
|
17
|
+
* allows/denies the tool call. The compiled markdown's frontmatter is the
|
|
18
|
+
* single source of truth — the same list that documents intent IS the list the
|
|
19
|
+
* hook enforces, so the two agree by construction (see `enforcedTools`).
|
|
20
|
+
*
|
|
21
|
+
* Which agent is active is recorded in `.vigiles/active-agent.json` — Claude
|
|
22
|
+
* Code hooks don't surface the dispatched subagent, so vigiles records it
|
|
23
|
+
* (mirrors `.vigiles/active-skill.json`). The decision logic below is
|
|
24
|
+
* harness-agnostic and fully testable.
|
|
25
|
+
*/
|
|
26
|
+
Object.defineProperty(exports, "__esModule", { value: true });
|
|
27
|
+
exports.parseAgentTools = parseAgentTools;
|
|
28
|
+
exports.decidePreToolUse = decidePreToolUse;
|
|
29
|
+
exports.setActiveAgent = setActiveAgent;
|
|
30
|
+
exports.clearActiveAgent = clearActiveAgent;
|
|
31
|
+
exports.readActiveAgent = readActiveAgent;
|
|
32
|
+
exports.evaluatePreToolUse = evaluatePreToolUse;
|
|
33
|
+
const node_fs_1 = require("node:fs");
|
|
34
|
+
const node_path_1 = require("node:path");
|
|
35
|
+
// ---------------------------------------------------------------------------
|
|
36
|
+
// Parse the tool contract from a compiled agent .md
|
|
37
|
+
// ---------------------------------------------------------------------------
|
|
38
|
+
/** Extract the YAML frontmatter block (between the first pair of `---` fences). */
|
|
39
|
+
function extractFrontmatter(markdown) {
|
|
40
|
+
const lines = markdown.split("\n");
|
|
41
|
+
let start = -1;
|
|
42
|
+
for (let i = 0; i < lines.length; i++) {
|
|
43
|
+
if (lines[i].trim() === "---") {
|
|
44
|
+
start = i;
|
|
45
|
+
break;
|
|
46
|
+
}
|
|
47
|
+
}
|
|
48
|
+
if (start === -1)
|
|
49
|
+
return null;
|
|
50
|
+
for (let i = start + 1; i < lines.length; i++) {
|
|
51
|
+
if (lines[i].trim() === "---") {
|
|
52
|
+
return lines.slice(start + 1, i).join("\n");
|
|
53
|
+
}
|
|
54
|
+
}
|
|
55
|
+
return null;
|
|
56
|
+
}
|
|
57
|
+
/**
|
|
58
|
+
* Parse an agent's allowed-tools contract from its compiled markdown.
|
|
59
|
+
*
|
|
60
|
+
* Returns the list of allowed tool names, or `null` when the agent declares no
|
|
61
|
+
* `tools:` line at all — which in Claude Code means it inherits EVERY tool (the
|
|
62
|
+
* #1 footgun). `null` is the "no restriction" signal the decision logic honors;
|
|
63
|
+
* an empty list (`tools:` with nothing after it) means "no tools allowed".
|
|
64
|
+
*/
|
|
65
|
+
function parseAgentTools(markdown) {
|
|
66
|
+
const fm = extractFrontmatter(markdown);
|
|
67
|
+
if (fm === null)
|
|
68
|
+
return null;
|
|
69
|
+
const match = /^tools:[ \t]*(.*)$/m.exec(fm);
|
|
70
|
+
if (!match)
|
|
71
|
+
return null;
|
|
72
|
+
return match[1]
|
|
73
|
+
.split(",")
|
|
74
|
+
.map((t) => t.trim())
|
|
75
|
+
.filter((t) => t.length > 0);
|
|
76
|
+
}
|
|
77
|
+
/**
|
|
78
|
+
* Decide whether `tool` is allowed under an agent's tool contract. Pure, so the
|
|
79
|
+
* rail is unit-testable without spawning anything.
|
|
80
|
+
*
|
|
81
|
+
* - `allowed === null` → the agent declared no `tools:` line, so it inherits
|
|
82
|
+
* everything and the rail imposes no restriction (allow).
|
|
83
|
+
* - otherwise → allow iff the tool is in the allowlist; deny anything else,
|
|
84
|
+
* feeding the contract back to the model so it self-corrects.
|
|
85
|
+
*/
|
|
86
|
+
function decidePreToolUse(allowed, tool) {
|
|
87
|
+
if (allowed === null)
|
|
88
|
+
return { allow: true, message: "" };
|
|
89
|
+
if (allowed.includes(tool))
|
|
90
|
+
return { allow: true, message: "" };
|
|
91
|
+
const list = allowed.length > 0 ? allowed.join(", ") : "(none)";
|
|
92
|
+
return {
|
|
93
|
+
allow: false,
|
|
94
|
+
message: `Tool "${tool}" is not in this subagent's allowed-tools contract ` +
|
|
95
|
+
`(${list}). Use only the listed tools, or widen the agent's \`tools\`.`,
|
|
96
|
+
};
|
|
97
|
+
}
|
|
98
|
+
// ---------------------------------------------------------------------------
|
|
99
|
+
// Active-agent tracking (mirrors .vigiles/active-skill.json)
|
|
100
|
+
// ---------------------------------------------------------------------------
|
|
101
|
+
const ACTIVE_PATH = ".vigiles/active-agent.json";
|
|
102
|
+
/** Record the subagent currently dispatched, so PreToolUse enforces its contract. */
|
|
103
|
+
function setActiveAgent(cwd, agentPath) {
|
|
104
|
+
const p = (0, node_path_1.resolve)(cwd, ACTIVE_PATH);
|
|
105
|
+
(0, node_fs_1.mkdirSync)((0, node_path_1.dirname)(p), { recursive: true });
|
|
106
|
+
(0, node_fs_1.writeFileSync)(p, JSON.stringify({ agent: agentPath }) + "\n");
|
|
107
|
+
}
|
|
108
|
+
/** Clear the active-agent marker (the subagent finished). */
|
|
109
|
+
function clearActiveAgent(cwd) {
|
|
110
|
+
const p = (0, node_path_1.resolve)(cwd, ACTIVE_PATH);
|
|
111
|
+
if ((0, node_fs_1.existsSync)(p))
|
|
112
|
+
(0, node_fs_1.rmSync)(p);
|
|
113
|
+
}
|
|
114
|
+
/** The path of the active agent's compiled `.md`, or null when none is active. */
|
|
115
|
+
function readActiveAgent(cwd) {
|
|
116
|
+
const p = (0, node_path_1.resolve)(cwd, ACTIVE_PATH);
|
|
117
|
+
if (!(0, node_fs_1.existsSync)(p))
|
|
118
|
+
return null;
|
|
119
|
+
try {
|
|
120
|
+
const parsed = JSON.parse((0, node_fs_1.readFileSync)(p, "utf-8"));
|
|
121
|
+
return typeof parsed.agent === "string" ? parsed.agent : null;
|
|
122
|
+
}
|
|
123
|
+
catch {
|
|
124
|
+
return null;
|
|
125
|
+
}
|
|
126
|
+
}
|
|
127
|
+
// ---------------------------------------------------------------------------
|
|
128
|
+
// PreToolUse-hook decision
|
|
129
|
+
// ---------------------------------------------------------------------------
|
|
130
|
+
/**
|
|
131
|
+
* PreToolUse-hook decision. If an agent is active, parse its compiled `.md`
|
|
132
|
+
* tool contract and allow the call only when the tool is in the allowlist;
|
|
133
|
+
* otherwise block and tell the model which tools it may use. With no active
|
|
134
|
+
* agent (or an agent that inherits all tools), always allow — the rail only
|
|
135
|
+
* constrains agents that declared a contract.
|
|
136
|
+
*/
|
|
137
|
+
function evaluatePreToolUse(cwd, tool) {
|
|
138
|
+
const agentPath = readActiveAgent(cwd);
|
|
139
|
+
if (!agentPath)
|
|
140
|
+
return { allow: true, message: "" };
|
|
141
|
+
const full = (0, node_path_1.resolve)(cwd, agentPath);
|
|
142
|
+
if (!(0, node_fs_1.existsSync)(full))
|
|
143
|
+
return { allow: true, message: "" };
|
|
144
|
+
const allowed = parseAgentTools((0, node_fs_1.readFileSync)(full, "utf-8"));
|
|
145
|
+
return decidePreToolUse(allowed, tool);
|
|
146
|
+
}
|
|
147
|
+
//# sourceMappingURL=agent-runtime.js.map
|
package/dist/cli.js
CHANGED
|
@@ -23,6 +23,7 @@ const frontmatter_js_1 = require("./frontmatter.js");
|
|
|
23
23
|
const generate_schema_js_1 = require("./generate-schema.js");
|
|
24
24
|
const compile_generator_js_1 = require("./compile-generator.js");
|
|
25
25
|
const action_gate_js_1 = require("./action-gate.js");
|
|
26
|
+
const agent_runtime_js_1 = require("./agent-runtime.js");
|
|
26
27
|
const refs_js_1 = require("./refs.js");
|
|
27
28
|
const mcp_js_1 = require("./mcp.js");
|
|
28
29
|
const skill_runtime_js_1 = require("./skill-runtime.js");
|
|
@@ -177,8 +178,57 @@ function compileSkillToFile(spec, specPath) {
|
|
|
177
178
|
printErrors(specPath, errors);
|
|
178
179
|
return false;
|
|
179
180
|
}
|
|
181
|
+
/** Compile a subagent spec → agents/<name>.md (with its result-contract section). */
|
|
182
|
+
function compileAgentToFile(spec, specPath) {
|
|
183
|
+
const outputPath = specPath.replace(/\.spec\.ts$/, "");
|
|
184
|
+
const { markdown, errors } = (0, compile_js_1.compileAgent)(spec, {
|
|
185
|
+
basePath: process.cwd(),
|
|
186
|
+
specFile: specPath,
|
|
187
|
+
});
|
|
188
|
+
(0, node_fs_1.writeFileSync)((0, node_path_1.resolve)(process.cwd(), outputPath), markdown);
|
|
189
|
+
if (errors.length === 0) {
|
|
190
|
+
console.log(`\n✓ ${specPath} → ${outputPath}`);
|
|
191
|
+
return true;
|
|
192
|
+
}
|
|
193
|
+
console.log(`\n✗ ${specPath} — ${String(errors.length)} error(s)`);
|
|
194
|
+
printErrors(specPath, errors);
|
|
195
|
+
return false;
|
|
196
|
+
}
|
|
197
|
+
/**
|
|
198
|
+
* Compile a railway spec → the orchestrator command markdown. `knownAgents` is
|
|
199
|
+
* the set of compiled agent names in the project, so every `delegate()` target
|
|
200
|
+
* is resolved at compile time (an unknown target is a stale-ref error).
|
|
201
|
+
*/
|
|
202
|
+
function compileRailwayToFile(spec, specPath, knownAgents) {
|
|
203
|
+
const outputPath = specPath.replace(/\.spec\.ts$/, "");
|
|
204
|
+
const { markdown, errors } = (0, compile_js_1.compileRailway)(spec, {
|
|
205
|
+
specFile: specPath,
|
|
206
|
+
knownAgents,
|
|
207
|
+
});
|
|
208
|
+
(0, node_fs_1.writeFileSync)((0, node_path_1.resolve)(process.cwd(), outputPath), markdown);
|
|
209
|
+
if (errors.length === 0) {
|
|
210
|
+
console.log(`\n✓ ${specPath} → ${outputPath}`);
|
|
211
|
+
return true;
|
|
212
|
+
}
|
|
213
|
+
console.log(`\n✗ ${specPath} — ${String(errors.length)} error(s)`);
|
|
214
|
+
printErrors(specPath, errors);
|
|
215
|
+
return false;
|
|
216
|
+
}
|
|
217
|
+
/** Names of every compiled agent spec in the project — resolves delegate() targets. */
|
|
218
|
+
async function collectAgentNames() {
|
|
219
|
+
const names = [];
|
|
220
|
+
for (const p of findSpecs()) {
|
|
221
|
+
const s = await loadSpec(p);
|
|
222
|
+
if (s && s._specType === "agent")
|
|
223
|
+
names.push(s.name);
|
|
224
|
+
}
|
|
225
|
+
return names;
|
|
226
|
+
}
|
|
180
227
|
async function compile(specPaths, config) {
|
|
181
228
|
let allValid = true;
|
|
229
|
+
// Resolved lazily on the first railway spec — every delegate() target is
|
|
230
|
+
// checked against the agents defined anywhere in the project.
|
|
231
|
+
let knownAgents = null;
|
|
182
232
|
for (const specPath of specPaths) {
|
|
183
233
|
// Generator skills can't be executed to markdown — compile from source.
|
|
184
234
|
const source = (0, node_fs_1.readFileSync)((0, node_path_1.resolve)(process.cwd(), specPath), "utf-8");
|
|
@@ -202,6 +252,15 @@ async function compile(specPaths, config) {
|
|
|
202
252
|
if (!compileSkillToFile(spec, specPath))
|
|
203
253
|
allValid = false;
|
|
204
254
|
}
|
|
255
|
+
else if (spec._specType === "agent") {
|
|
256
|
+
if (!compileAgentToFile(spec, specPath))
|
|
257
|
+
allValid = false;
|
|
258
|
+
}
|
|
259
|
+
else if (spec._specType === "railway") {
|
|
260
|
+
knownAgents ??= await collectAgentNames();
|
|
261
|
+
if (!compileRailwayToFile(spec, specPath, knownAgents))
|
|
262
|
+
allValid = false;
|
|
263
|
+
}
|
|
205
264
|
}
|
|
206
265
|
return allValid;
|
|
207
266
|
}
|
|
@@ -1612,6 +1671,44 @@ function skillStartCommand(target) {
|
|
|
1612
1671
|
(0, skill_runtime_js_1.setActiveSkill)(process.cwd(), target);
|
|
1613
1672
|
console.log(`Active skill: ${target}`);
|
|
1614
1673
|
}
|
|
1674
|
+
/**
|
|
1675
|
+
* PreToolUse-hook entrypoint: enforce the active subagent's allowed-tools
|
|
1676
|
+
* contract. Reads the tool event on stdin, parses the active agent's compiled
|
|
1677
|
+
* `.md` tool rail, and blocks (exit 2 + reason on stderr) any tool outside it —
|
|
1678
|
+
* the deterministic boundary `tools:` alone can't provide (Claude Code #54898).
|
|
1679
|
+
*/
|
|
1680
|
+
function agentHookCommand() {
|
|
1681
|
+
let raw = "";
|
|
1682
|
+
try {
|
|
1683
|
+
raw = (0, node_fs_1.readFileSync)(0, "utf-8");
|
|
1684
|
+
}
|
|
1685
|
+
catch {
|
|
1686
|
+
/* no stdin */
|
|
1687
|
+
}
|
|
1688
|
+
let tool = "";
|
|
1689
|
+
try {
|
|
1690
|
+
tool = JSON.parse(raw).tool_name ?? "";
|
|
1691
|
+
}
|
|
1692
|
+
catch {
|
|
1693
|
+
/* malformed input → no tool, allow */
|
|
1694
|
+
}
|
|
1695
|
+
if (!tool)
|
|
1696
|
+
return;
|
|
1697
|
+
const decision = (0, agent_runtime_js_1.evaluatePreToolUse)(process.cwd(), tool);
|
|
1698
|
+
if (!decision.allow) {
|
|
1699
|
+
console.error(decision.message);
|
|
1700
|
+
process.exit(2);
|
|
1701
|
+
}
|
|
1702
|
+
}
|
|
1703
|
+
/** Mark a subagent active so the PreToolUse hook enforces its tool contract. */
|
|
1704
|
+
function agentStartCommand(target) {
|
|
1705
|
+
if (!target) {
|
|
1706
|
+
console.error("Usage: vigiles agent-start <agents/<name>.md>");
|
|
1707
|
+
process.exit(2);
|
|
1708
|
+
}
|
|
1709
|
+
(0, agent_runtime_js_1.setActiveAgent)(process.cwd(), target);
|
|
1710
|
+
console.log(`Active agent: ${target}`);
|
|
1711
|
+
}
|
|
1615
1712
|
/** Dispatch the skill-runtime subcommands. Returns false if unrecognized. */
|
|
1616
1713
|
function handleSkillCommand(command, restArgs) {
|
|
1617
1714
|
switch (command) {
|
|
@@ -1627,6 +1724,15 @@ function handleSkillCommand(command, restArgs) {
|
|
|
1627
1724
|
case "skill-hook":
|
|
1628
1725
|
skillHookCommand();
|
|
1629
1726
|
return true;
|
|
1727
|
+
case "agent-start":
|
|
1728
|
+
agentStartCommand(restArgs[0]);
|
|
1729
|
+
return true;
|
|
1730
|
+
case "agent-done":
|
|
1731
|
+
(0, agent_runtime_js_1.clearActiveAgent)(process.cwd());
|
|
1732
|
+
return true;
|
|
1733
|
+
case "agent-hook":
|
|
1734
|
+
agentHookCommand();
|
|
1735
|
+
return true;
|
|
1630
1736
|
case "action-hook":
|
|
1631
1737
|
actionHookCommand();
|
|
1632
1738
|
return true;
|
package/dist/compile.d.ts
CHANGED
|
@@ -4,7 +4,7 @@
|
|
|
4
4
|
* Reads .spec.ts files, validates references, and produces
|
|
5
5
|
* markdown instruction files with integrity hashes.
|
|
6
6
|
*/
|
|
7
|
-
import type { ClaudeSpec, SkillSpec } from "./spec.js";
|
|
7
|
+
import type { ClaudeSpec, SkillSpec, AgentSpec, Railway } from "./spec.js";
|
|
8
8
|
import type { LinterCheckResult } from "./linters.js";
|
|
9
9
|
/** @internal Compute SHA-256 hash of content (excluding any existing hash line). */
|
|
10
10
|
export declare function computeHash(content: string): string;
|
|
@@ -24,7 +24,7 @@ export declare function verifyHash(content: string): {
|
|
|
24
24
|
*/
|
|
25
25
|
/** @internal */ export declare function estimateTokens(text: string): number;
|
|
26
26
|
export interface CompileError {
|
|
27
|
-
type: "stale-file" | "stale-command" | "stale-ref" | "invalid-rule" | "budget-exceeded" | "section-too-long" | "section-has-header" | "reserved-section-key" | "spec-name-mismatch";
|
|
27
|
+
type: "stale-file" | "stale-command" | "stale-ref" | "invalid-rule" | "budget-exceeded" | "section-too-long" | "section-has-header" | "reserved-section-key" | "spec-name-mismatch" | "unknown-tool" | "invalid-railway";
|
|
28
28
|
message: string;
|
|
29
29
|
path?: string;
|
|
30
30
|
}
|
|
@@ -79,6 +79,35 @@ export declare function compileSkill(spec: SkillSpec, options?: {
|
|
|
79
79
|
basePath?: string;
|
|
80
80
|
specFile?: string;
|
|
81
81
|
}): CompileSkillResult;
|
|
82
|
+
export interface CompileAgentResult {
|
|
83
|
+
markdown: string;
|
|
84
|
+
errors: CompileError[];
|
|
85
|
+
}
|
|
86
|
+
/**
|
|
87
|
+
* Compile an AgentSpec into a subagent markdown file with YAML frontmatter.
|
|
88
|
+
* Verifies the tool contract and the body's references; the marks the body
|
|
89
|
+
* carries (`vigiles:symbol`, file/cmd refs) are the same ones `audit` re-checks.
|
|
90
|
+
*/
|
|
91
|
+
export declare function compileAgent(spec: AgentSpec, options?: {
|
|
92
|
+
basePath?: string;
|
|
93
|
+
specFile?: string;
|
|
94
|
+
}): CompileAgentResult;
|
|
95
|
+
export interface CompileRailwayOptions {
|
|
96
|
+
/** Names of compiled agents, to resolve `delegate` targets. Skipped if omitted. */
|
|
97
|
+
knownAgents?: readonly string[];
|
|
98
|
+
specFile?: string;
|
|
99
|
+
}
|
|
100
|
+
export interface CompileRailwayResult {
|
|
101
|
+
markdown: string;
|
|
102
|
+
errors: CompileError[];
|
|
103
|
+
}
|
|
104
|
+
/** Verify a railway: non-empty, bounded recovery, every delegate target real. */
|
|
105
|
+
export declare function validateRailway(rw: Railway, knownAgents?: readonly string[]): CompileError[];
|
|
106
|
+
/**
|
|
107
|
+
* Compile a railway into an orchestrator command markdown (with integrity hash),
|
|
108
|
+
* resolving every delegate target against `knownAgents` when provided.
|
|
109
|
+
*/
|
|
110
|
+
export declare function compileRailway(rw: Railway, options?: CompileRailwayOptions): CompileRailwayResult;
|
|
82
111
|
export interface HashCheckResult {
|
|
83
112
|
hasHash: boolean;
|
|
84
113
|
valid: boolean;
|
|
@@ -101,5 +130,5 @@ export interface AdoptResult {
|
|
|
101
130
|
* Compare a generated file against what the spec would produce.
|
|
102
131
|
* Returns the diff so users can see what was manually changed.
|
|
103
132
|
*/
|
|
104
|
-
export declare function adoptDiff(filePath: string, spec: ClaudeSpec | SkillSpec, basePath: string): AdoptResult;
|
|
133
|
+
export declare function adoptDiff(filePath: string, spec: ClaudeSpec | SkillSpec | AgentSpec, basePath: string): AdoptResult;
|
|
105
134
|
//# sourceMappingURL=compile.d.ts.map
|