vigiles 4.1.0 → 5.0.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +71 -79
- package/dist/adapters/claude-code/agent-runtime.d.ts +1 -1
- package/dist/adapters/claude-code/agent-runtime.js +1 -1
- package/dist/arg-match.d.ts +28 -0
- package/dist/arg-match.js +61 -0
- package/dist/check.d.ts +52 -1
- package/dist/check.js +121 -0
- package/dist/claude-code.d.ts +1 -0
- package/dist/claude-code.js +11 -0
- package/dist/cli.js +86 -14
- package/dist/core/types.d.ts +17 -12
- package/dist/core/validate.js +14 -3
- package/dist/eval-cache.d.ts +49 -1
- package/dist/eval-cache.js +84 -4
- package/dist/eval.d.ts +243 -14
- package/dist/eval.js +540 -28
- package/dist/integration.d.ts +15 -10
- package/dist/integration.js +27 -10
- package/dist/linting.d.ts +4 -3
- package/dist/linting.js +4 -3
- package/dist/setup-plan.js +3 -1
- package/dist/test-coverage.d.ts +11 -12
- package/dist/test-coverage.js +14 -19
- package/dist/testing.d.ts +4 -1
- package/dist/testing.js +24 -2
- package/dist/tool-intercept.d.ts +101 -0
- package/dist/tool-intercept.js +165 -0
- package/dist/tool-stub.d.ts +35 -0
- package/dist/tool-stub.js +92 -0
- package/package.json +36 -19
- package/skills/test-harness/SKILL.md +56 -3
package/README.md
CHANGED
|
@@ -5,12 +5,11 @@
|
|
|
5
5
|
<h1 align="center">vigiles</h1>
|
|
6
6
|
|
|
7
7
|
<p align="center">
|
|
8
|
-
<
|
|
8
|
+
<strong>Lint & test the harness your AI agent runs on.</strong>
|
|
9
9
|
</p>
|
|
10
10
|
|
|
11
11
|
<p align="center">
|
|
12
|
-
<
|
|
13
|
-
vigiles <strong>lints</strong> the references your instruction files make — linter rules, file paths, scripts, code symbols — and <strong>tests</strong> whether your hooks, skills, and CLAUDE.md actually change what the agent does.
|
|
12
|
+
Your CLAUDE.md, hooks, and skills steer the agent — but nothing checks they're <em>true</em>, and nothing tests they <em>work</em>. vigiles does both.
|
|
14
13
|
</p>
|
|
15
14
|
|
|
16
15
|
<p align="center">
|
|
@@ -22,81 +21,17 @@
|
|
|
22
21
|
---
|
|
23
22
|
|
|
24
23
|
`Agent = Model + Harness`. You'd never ship an app without a linter and a test
|
|
25
|
-
suite — yet
|
|
26
|
-
deterministic layer for
|
|
27
|
-
files make and **tests** that your hooks and skills actually fire. Two independent
|
|
28
|
-
pillars — adopt either, or both:
|
|
24
|
+
suite — yet the harness steering your agent runs on vibes. vigiles[^name] is the
|
|
25
|
+
deterministic layer for it, and does two independent things — adopt either, or both:
|
|
29
26
|
|
|
30
|
-
|
|
|
31
|
-
|
|
|
32
|
-
|
|
|
33
|
-
|
|
|
27
|
+
| | |
|
|
28
|
+
| ----------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
|
|
29
|
+
| **🔎 Lint** | Every file path, script, code symbol, and linter rule your CLAUDE.md cites is checked against reality — so a renamed file or a disabled rule can't silently mislead the agent. **[→](docs/verifying-instruction-files.md)** |
|
|
30
|
+
| **🧪 Test** | Hooks, skills, and subagents are code. vigiles tests they _do their job_ — and almost all of it is **deterministic, no API key**; the real-model evals run on your **Claude subscription**, not metered tokens. **[→](docs/harness-testing.md)** |
|
|
34
31
|
|
|
35
|
-
|
|
36
|
-
|
|
37
|
-
|
|
38
|
-
|
|
39
|
-
## ① Lint — your CLAUDE.md lies to your agent
|
|
40
|
-
|
|
41
|
-
Your CLAUDE.md says _"enforce `eslint/no-console`."_ But it was switched off
|
|
42
|
-
months ago — and the agent trusts the claim. (Same story for the file path it
|
|
43
|
-
cites that got renamed, and the script that was deleted.)
|
|
44
|
-
|
|
45
|
-
**Without vigiles:** nobody checks. The agent acts on fiction.
|
|
46
|
-
|
|
47
|
-
**With vigiles:** `npx vigiles lint` resolves every reference against reality —
|
|
48
|
-
|
|
49
|
-
```text
|
|
50
|
-
CLAUDE.md (inline mode):
|
|
51
|
-
✗ line 1: Rule "eslint/no-console" exists but is disabled in eslint config
|
|
52
|
-
✓ line 2: eslint/eqeqeq
|
|
53
|
-
✗ line 3: Rule "no-consoel" not found in eslint. Did you mean: "eslint/no-console"?
|
|
54
|
-
```
|
|
55
|
-
|
|
56
|
-
It resolves rule names across **7 linter catalogs** — the rule exists **and is
|
|
57
|
-
enabled** — and checks file paths, scripts, and code symbols the same way. Start
|
|
58
|
-
with one comment, no new files:
|
|
59
|
-
|
|
60
|
-
```md
|
|
61
|
-
<!-- vigiles:enforce eslint/no-console "Route output through logger.ts" -->
|
|
62
|
-
```
|
|
63
|
-
|
|
64
|
-
Step up to a typed `.spec.ts` (compiled to CLAUDE.md, compiler-grade) when you
|
|
65
|
-
want it. **[Full guide →](docs/verifying-instruction-files.md)**
|
|
66
|
-
|
|
67
|
-
## ② Test — does your harness actually fire?
|
|
68
|
-
|
|
69
|
-
A hook can be wired wrong, a skill's description can fail to trigger, injected
|
|
70
|
-
context can never reach the model — silently, all passing a naive "did it run?"
|
|
71
|
-
check.
|
|
72
|
-
|
|
73
|
-
**Without vigiles:** you assume your `--no-verify` guard blocks. You don't know.
|
|
74
|
-
|
|
75
|
-
**With vigiles:** a deterministic test proves it — no model, no API key,
|
|
76
|
-
milliseconds:
|
|
77
|
-
|
|
78
|
-
```typescript
|
|
79
|
-
import { runHook } from "vigiles/testing";
|
|
80
|
-
|
|
81
|
-
const r = runHook(guard, {
|
|
82
|
-
hook_event_name: "PreToolUse",
|
|
83
|
-
tool_name: "Bash",
|
|
84
|
-
tool_input: { command: "git commit --no-verify" },
|
|
85
|
-
});
|
|
86
|
-
assert(r.blocked); // a red ✗ here means your hook silently lets it through
|
|
87
|
-
```
|
|
88
|
-
|
|
89
|
-
```text
|
|
90
|
-
✓ guard blocks --no-verify and allows a clean commit
|
|
91
|
-
|
|
92
|
-
2 passed.
|
|
93
|
-
```
|
|
94
|
-
|
|
95
|
-
Three tiers, cheapest first: **`runHook`** (a hook's logic), **`runHarnessTest`**
|
|
96
|
-
(the real agent CLI against a scripted mock model), **`runEval`** (the real model
|
|
97
|
-
A/B with a significance gate). **Testing a skill?** `measureTriggerRate` checks
|
|
98
|
-
its description actually **fires** across varied prompts (recall) without
|
|
99
|
-
hijacking unrelated ones (precision). **[Full guide →](docs/harness-testing.md)**
|
|
32
|
+
Pick the one that hurts today. **Works with Claude Code and Codex**
|
|
33
|
+
([`vigiles/codex`](docs/harnesses.md)) behind a five-port adapter;
|
|
34
|
+
[custom adapters welcome](docs/authoring-an-adapter.md).
|
|
100
35
|
|
|
101
36
|
## Quick start
|
|
102
37
|
|
|
@@ -105,7 +40,7 @@ hijacking unrelated ones (precision). **[Full guide →](docs/harness-testing.md
|
|
|
105
40
|
```text
|
|
106
41
|
Install vigiles in this repo and run it. Verify my CLAUDE.md / AGENTS.md
|
|
107
42
|
references and show me what's stale, then write and run a harness test for one
|
|
108
|
-
of my hooks or skills. Use good defaults (
|
|
43
|
+
of my hooks or skills. Use good defaults (lint + test, non-interactive), but
|
|
109
44
|
ask me first whether to gate it in CI, whether to add a real-model eval, and
|
|
110
45
|
whether to enforce strictly (--strict).
|
|
111
46
|
```
|
|
@@ -113,7 +48,7 @@ whether to enforce strictly (--strict).
|
|
|
113
48
|
Or do it yourself:
|
|
114
49
|
|
|
115
50
|
```bash
|
|
116
|
-
npx vigiles init # sets up
|
|
51
|
+
npx vigiles init # sets up lint + test: spec + harness test + CI + plugin
|
|
117
52
|
```
|
|
118
53
|
|
|
119
54
|
It's interactive in a terminal and non-interactive for agents/CI (or with
|
|
@@ -124,7 +59,7 @@ tell your agent _"test my skills"_ and it picks the tier and writes the test.
|
|
|
124
59
|
<details>
|
|
125
60
|
<summary>What <code>init</code> sets up</summary>
|
|
126
61
|
|
|
127
|
-
- **Both
|
|
62
|
+
- **Both lint and test** by default; scope with `--lint` / `--test` (one or both).
|
|
128
63
|
- Adds `vigiles` to your `devDependencies`.
|
|
129
64
|
- Installs the Claude Code plugin (skills + hooks) via the marketplace —
|
|
130
65
|
globally, never vendored into your repo.
|
|
@@ -140,6 +75,61 @@ Prefer to write tests yourself? They can be JS **or** TS
|
|
|
140
75
|
|
|
141
76
|
</details>
|
|
142
77
|
|
|
78
|
+
## ① Lint — your CLAUDE.md lies to your agent
|
|
79
|
+
|
|
80
|
+
Your CLAUDE.md points the agent at `src/auth/login.ts` and tells it to run
|
|
81
|
+
`npm run check`. But the file moved to `src/auth/session.ts` six commits ago, and
|
|
82
|
+
the script was renamed. The agent trusts the stale claim and acts on fiction.
|
|
83
|
+
|
|
84
|
+
`npx vigiles lint` resolves every reference against reality:
|
|
85
|
+
|
|
86
|
+
```text
|
|
87
|
+
CLAUDE.md:
|
|
88
|
+
✗ src/auth/login.ts — no such file (renamed or moved?)
|
|
89
|
+
✗ npm run check — not in package.json. Did you mean: "check:types"?
|
|
90
|
+
✓ @typescript-eslint/no-floating-promises — exists and enabled in eslint config
|
|
91
|
+
```
|
|
92
|
+
|
|
93
|
+
File paths, scripts, and code symbols — plus linter rules across **7 catalogs**
|
|
94
|
+
(the rule exists **and is enabled**). Start with one inline comment, no new files;
|
|
95
|
+
step up to a typed `.spec.ts` (compiled to CLAUDE.md, compiler-grade) when you want
|
|
96
|
+
it. **[Full guide →](docs/verifying-instruction-files.md)**
|
|
97
|
+
|
|
98
|
+
## ② Test — does your harness do its job?
|
|
99
|
+
|
|
100
|
+
A hook can be wired wrong. A skill's description can fail to trigger — or hijack
|
|
101
|
+
unrelated prompts. Injected context can never reach the model. All of it passes a
|
|
102
|
+
naive "did it run?" check. vigiles tests the assembled harness for real:
|
|
103
|
+
|
|
104
|
+
```typescript
|
|
105
|
+
import { runHook } from "vigiles/testing";
|
|
106
|
+
|
|
107
|
+
const r = runHook(guard, {
|
|
108
|
+
hook_event_name: "PreToolUse",
|
|
109
|
+
tool_name: "Bash",
|
|
110
|
+
tool_input: { command: "git commit --no-verify" },
|
|
111
|
+
});
|
|
112
|
+
assert(r.blocked); // a red ✗ means your guard silently lets it through
|
|
113
|
+
```
|
|
114
|
+
|
|
115
|
+
It goes well past _"did it fire?"_:
|
|
116
|
+
|
|
117
|
+
- **Hooks block** what they must — `runHook`, or the real agent CLI via `runHarnessTest`.
|
|
118
|
+
- **Skills trigger** on the right prompts and stay quiet on the wrong ones — recall _and_ precision (`measureTriggerRate`).
|
|
119
|
+
- **Behaviour is good** — score a skill's output directly, or A/B it on-vs-off for the real lift over no-skill (`measure` / `runEval`, with significance testing).
|
|
120
|
+
- **Safety holds** — the agent _didn't_ push to the wrong branch or hit a paid API; `interceptTools` catches the attempt so the side effect never happens.
|
|
121
|
+
|
|
122
|
+
**The eval you can actually afford.** Almost every tier runs with **no model and
|
|
123
|
+
no API key** — milliseconds, on every commit. The rest drive your own `claude` CLI:
|
|
124
|
+
|
|
125
|
+
| | Runs on | Cost |
|
|
126
|
+
| ---------------------- | ----------------------- | ------------------------------------------- |
|
|
127
|
+
| promptfoo, DeepEval, … | metered API SDK | billed **per token, every run** |
|
|
128
|
+
| **vigiles** | your Claude Pro/Max sub | **$0 extra** — and most tiers need no model |
|
|
129
|
+
|
|
130
|
+
That's why you can eval your harness on every change, not just once.
|
|
131
|
+
**[How it works →](docs/harness-testing.md)** · **[Why it's affordable →](docs/eval-architecture.md)** · **[Safety model →](docs/safety.md)**
|
|
132
|
+
|
|
143
133
|
## More
|
|
144
134
|
|
|
145
135
|
- **[CLI & GitHub Action →](docs/cli.md)** — every command, the Action (inputs / output / versioning), the Claude Code plugin, and the five `lint` rules.
|
|
@@ -150,3 +140,5 @@ Prefer to write tests yourself? They can be JS **or** TS
|
|
|
150
140
|
## License
|
|
151
141
|
|
|
152
142
|
[MIT](LICENSE)
|
|
143
|
+
|
|
144
|
+
[^name]: **vigiles** — the watchmen of ancient Rome, who guarded the city (and fought its fires) by night. _Quis custodiet ipsos custodes?_ — "who watches the watchmen?" (Juvenal, _Satire VI_).
|
|
@@ -3,7 +3,7 @@
|
|
|
3
3
|
*
|
|
4
4
|
* A subagent declares an allowed-tools contract in its frontmatter (`tools:`).
|
|
5
5
|
* But that field is documentation, not a hard runtime boundary (Claude Code
|
|
6
|
-
*
|
|
6
|
+
* #4740/#21460, SDK #172): permissions are session-wide, a subagent inherits the parent
|
|
7
7
|
* session's grants, and `tools:` only filters what's *offered* — it can't deny
|
|
8
8
|
* what the session allows. The deterministic layer that actually closes the gap
|
|
9
9
|
* is a **PreToolUse hook** that blocks any tool the active agent's contract
|
|
@@ -4,7 +4,7 @@
|
|
|
4
4
|
*
|
|
5
5
|
* A subagent declares an allowed-tools contract in its frontmatter (`tools:`).
|
|
6
6
|
* But that field is documentation, not a hard runtime boundary (Claude Code
|
|
7
|
-
*
|
|
7
|
+
* #4740/#21460, SDK #172): permissions are session-wide, a subagent inherits the parent
|
|
8
8
|
* session's grants, and `tools:` only filters what's *offered* — it can't deny
|
|
9
9
|
* what the session allows. The deterministic layer that actually closes the gap
|
|
10
10
|
* is a **PreToolUse hook** that blocks any tool the active agent's contract
|
|
@@ -0,0 +1,28 @@
|
|
|
1
|
+
/**
|
|
2
|
+
* `ArgMatcher` — a small, serializable matcher over a tool call's `input`.
|
|
3
|
+
*
|
|
4
|
+
* Shared by the check vocabulary (`toolWith` / `notTool` in `src/check.ts`) and
|
|
5
|
+
* the tool-interception seam (`src/tool-intercept.ts`), so "did the agent call
|
|
6
|
+
* this tool with these arguments?" means the same thing whether you're *asserting*
|
|
7
|
+
* on a captured call or *intercepting* one before it runs. Pure + model-free.
|
|
8
|
+
*/
|
|
9
|
+
/**
|
|
10
|
+
* A declarative matcher over a tool call's `input`, keyed by **dot-path** (e.g.
|
|
11
|
+
* `"body.prompt"`). Each value is matched against the value at that path: a
|
|
12
|
+
* `RegExp` is a pattern over the stringified value (use this for "contains"), and
|
|
13
|
+
* a `string`/`number`/`boolean` is an **exact** match (use this for "equals", e.g.
|
|
14
|
+
* a push target). All keys must match (AND). Serializable, so anything carrying
|
|
15
|
+
* one still round-trips through `toJSON`.
|
|
16
|
+
*/
|
|
17
|
+
export type ArgMatcher = Record<string, string | number | boolean | RegExp>;
|
|
18
|
+
/** Render any tool-input value as a string for matching / messages. */
|
|
19
|
+
export declare function stringifyValue(v: unknown): string;
|
|
20
|
+
/** Resolve a dot-path (`"a.b.c"`) within an arbitrary value, or undefined. */
|
|
21
|
+
export declare function getPath(obj: unknown, path: string): unknown;
|
|
22
|
+
/** Does `input` satisfy every entry of `matcher`? (RegExp = pattern, else exact.) */
|
|
23
|
+
export declare function matchesArgs(input: unknown, matcher: ArgMatcher): boolean;
|
|
24
|
+
/** A human-readable form of a matcher for failure messages. */
|
|
25
|
+
export declare function describeArgs(matcher: ArgMatcher): string;
|
|
26
|
+
/** Serialize a matcher for `toJSON` (RegExp → its string form). */
|
|
27
|
+
export declare function serializeArgs(matcher: ArgMatcher): Record<string, string | number | boolean>;
|
|
28
|
+
//# sourceMappingURL=arg-match.d.ts.map
|
|
@@ -0,0 +1,61 @@
|
|
|
1
|
+
"use strict";
|
|
2
|
+
/**
|
|
3
|
+
* `ArgMatcher` — a small, serializable matcher over a tool call's `input`.
|
|
4
|
+
*
|
|
5
|
+
* Shared by the check vocabulary (`toolWith` / `notTool` in `src/check.ts`) and
|
|
6
|
+
* the tool-interception seam (`src/tool-intercept.ts`), so "did the agent call
|
|
7
|
+
* this tool with these arguments?" means the same thing whether you're *asserting*
|
|
8
|
+
* on a captured call or *intercepting* one before it runs. Pure + model-free.
|
|
9
|
+
*/
|
|
10
|
+
Object.defineProperty(exports, "__esModule", { value: true });
|
|
11
|
+
exports.stringifyValue = stringifyValue;
|
|
12
|
+
exports.getPath = getPath;
|
|
13
|
+
exports.matchesArgs = matchesArgs;
|
|
14
|
+
exports.describeArgs = describeArgs;
|
|
15
|
+
exports.serializeArgs = serializeArgs;
|
|
16
|
+
/** Render any tool-input value as a string for matching / messages. */
|
|
17
|
+
function stringifyValue(v) {
|
|
18
|
+
if (typeof v === "string")
|
|
19
|
+
return v;
|
|
20
|
+
if (typeof v === "object" && v !== null) {
|
|
21
|
+
try {
|
|
22
|
+
return JSON.stringify(v) ?? "[object]";
|
|
23
|
+
}
|
|
24
|
+
catch {
|
|
25
|
+
return "[object]";
|
|
26
|
+
}
|
|
27
|
+
}
|
|
28
|
+
return String(v);
|
|
29
|
+
}
|
|
30
|
+
/** Resolve a dot-path (`"a.b.c"`) within an arbitrary value, or undefined. */
|
|
31
|
+
function getPath(obj, path) {
|
|
32
|
+
let cur = obj;
|
|
33
|
+
for (const part of path.split(".")) {
|
|
34
|
+
if (cur === null || typeof cur !== "object")
|
|
35
|
+
return undefined;
|
|
36
|
+
cur = cur[part];
|
|
37
|
+
}
|
|
38
|
+
return cur;
|
|
39
|
+
}
|
|
40
|
+
/** Does `input` satisfy every entry of `matcher`? (RegExp = pattern, else exact.) */
|
|
41
|
+
function matchesArgs(input, matcher) {
|
|
42
|
+
return Object.entries(matcher).every(([key, m]) => {
|
|
43
|
+
const value = getPath(input, key);
|
|
44
|
+
return m instanceof RegExp ? m.test(stringifyValue(value)) : value === m;
|
|
45
|
+
});
|
|
46
|
+
}
|
|
47
|
+
/** A human-readable form of a matcher for failure messages. */
|
|
48
|
+
function describeArgs(matcher) {
|
|
49
|
+
return Object.entries(matcher)
|
|
50
|
+
.map(([k, m]) => `${k}=${m instanceof RegExp ? String(m) : JSON.stringify(m)}`)
|
|
51
|
+
.join(", ");
|
|
52
|
+
}
|
|
53
|
+
/** Serialize a matcher for `toJSON` (RegExp → its string form). */
|
|
54
|
+
function serializeArgs(matcher) {
|
|
55
|
+
const out = {};
|
|
56
|
+
for (const [k, m] of Object.entries(matcher)) {
|
|
57
|
+
out[k] = m instanceof RegExp ? String(m) : m;
|
|
58
|
+
}
|
|
59
|
+
return out;
|
|
60
|
+
}
|
|
61
|
+
//# sourceMappingURL=arg-match.js.map
|
package/dist/check.d.ts
CHANGED
|
@@ -18,6 +18,8 @@
|
|
|
18
18
|
*/
|
|
19
19
|
import type { Trace } from "./harness-test.js";
|
|
20
20
|
import type { HookRunResult } from "./run-hook.js";
|
|
21
|
+
import { type ArgMatcher } from "./arg-match.js";
|
|
22
|
+
export type { ArgMatcher };
|
|
21
23
|
/** The outcome of evaluating one check against one result. */
|
|
22
24
|
export interface CheckResult {
|
|
23
25
|
/** Did the check hold? */
|
|
@@ -54,6 +56,28 @@ export declare function evalChecks<T>(target: T, checks: readonly Check<T>[]): C
|
|
|
54
56
|
export declare function assertChecks<T>(target: T, checks: readonly Check<T>[]): void;
|
|
55
57
|
/** The agent invoked a tool by this name (regardless of result). */
|
|
56
58
|
export declare function tool(name: string): Check<Trace>;
|
|
59
|
+
/**
|
|
60
|
+
* The agent used tool `name` with at least one call whose `input` matches `args`
|
|
61
|
+
* — the **argument** half of the tool-call spy. Asserts not just *that* a tool was
|
|
62
|
+
* reached but *how* it was called (the image-request body carried the style suffix,
|
|
63
|
+
* the panel spawn requested a non-expert), which a completion grader can't see.
|
|
64
|
+
*
|
|
65
|
+
* Cross-reference: ≈ promptfoo `is-valid-function-call` / DeepEval `ToolCorrectnessMetric`.
|
|
66
|
+
*/
|
|
67
|
+
export declare function toolWith(name: string, args: ArgMatcher): Check<Trace>;
|
|
68
|
+
/**
|
|
69
|
+
* The agent did NOT use tool `name` — the **safety / negative** assertion. With
|
|
70
|
+
* `args`, only calls whose `input` matches are forbidden (so "did not push to
|
|
71
|
+
* `main`" still allows pushing elsewhere; "no paid API call" forbids it outright).
|
|
72
|
+
* This is the highest-value, most-overlooked check, and the one a
|
|
73
|
+
* completion-grading eval structurally cannot make: it sees the agent's *decision*
|
|
74
|
+
* to act, not just its final text.
|
|
75
|
+
*
|
|
76
|
+
* Cross-reference: this negative/safety assertion of a *decision not to act* has
|
|
77
|
+
* no promptfoo / DeepEval / Inspect equivalent — they grade what the agent DID,
|
|
78
|
+
* not what it correctly refrained from doing.
|
|
79
|
+
*/
|
|
80
|
+
export declare function notTool(name: string, args?: ArgMatcher): Check<Trace>;
|
|
57
81
|
/** A skill resolved to this id (`<plugin>:<skill>`) without erroring. */
|
|
58
82
|
export declare function skill(id: string): Check<Trace>;
|
|
59
83
|
/** The agent's final output contains a substring / matches a RegExp. */
|
|
@@ -112,8 +136,13 @@ interface UsageTrace {
|
|
|
112
136
|
readonly usage: {
|
|
113
137
|
readonly costUsd: number;
|
|
114
138
|
readonly durationMs: number;
|
|
139
|
+
/** Fresh (uncached) input tokens, billed at full input price. */
|
|
115
140
|
readonly inputTokens: number;
|
|
116
141
|
readonly outputTokens: number;
|
|
142
|
+
/** Tokens written to the prompt cache this run (~1.25× input price). */
|
|
143
|
+
readonly cacheCreationTokens: number;
|
|
144
|
+
/** Tokens served from the prompt cache this run (~0.1× input price). */
|
|
145
|
+
readonly cacheReadTokens: number;
|
|
117
146
|
};
|
|
118
147
|
}
|
|
119
148
|
/** The run cost at most `maxUsd`. */
|
|
@@ -128,5 +157,27 @@ export declare function latency(opts: {
|
|
|
128
157
|
export declare function tokens(opts: {
|
|
129
158
|
max: number;
|
|
130
159
|
}): Check<UsageTrace>;
|
|
131
|
-
|
|
160
|
+
/**
|
|
161
|
+
* The run used at most `max` **fresh (uncached) input** tokens. The honest input
|
|
162
|
+
* side of a cost claim: a skill or CLAUDE.md injection adds input every turn, so
|
|
163
|
+
* a "compression" win on output can be erased here. (Cache reads are separate —
|
|
164
|
+
* see `cacheTokens`.)
|
|
165
|
+
*/
|
|
166
|
+
export declare function inputTokens(opts: {
|
|
167
|
+
max: number;
|
|
168
|
+
}): Check<UsageTrace>;
|
|
169
|
+
/** The run used at most `max` **output** tokens — the generation side. */
|
|
170
|
+
export declare function outputTokens(opts: {
|
|
171
|
+
max: number;
|
|
172
|
+
}): Check<UsageTrace>;
|
|
173
|
+
/**
|
|
174
|
+
* Bound the prompt-cache token classes a run uses. `maxCreation` caps tokens
|
|
175
|
+
* **written** to the cache (the ~1.25× write premium — a fresh/cold prompt);
|
|
176
|
+
* `maxRead` caps tokens **served** from cache (~0.1× input). Each constraint is
|
|
177
|
+
* checked only when provided; the check passes when every provided bound holds.
|
|
178
|
+
*/
|
|
179
|
+
export declare function cacheTokens(opts: {
|
|
180
|
+
maxCreation?: number;
|
|
181
|
+
maxRead?: number;
|
|
182
|
+
}): Check<UsageTrace>;
|
|
132
183
|
//# sourceMappingURL=check.d.ts.map
|
package/dist/check.js
CHANGED
|
@@ -3,6 +3,8 @@ Object.defineProperty(exports, "__esModule", { value: true });
|
|
|
3
3
|
exports.evalChecks = evalChecks;
|
|
4
4
|
exports.assertChecks = assertChecks;
|
|
5
5
|
exports.tool = tool;
|
|
6
|
+
exports.toolWith = toolWith;
|
|
7
|
+
exports.notTool = notTool;
|
|
6
8
|
exports.skill = skill;
|
|
7
9
|
exports.output = output;
|
|
8
10
|
exports.hookFired = hookFired;
|
|
@@ -17,7 +19,11 @@ exports.judged = judged;
|
|
|
17
19
|
exports.cost = cost;
|
|
18
20
|
exports.latency = latency;
|
|
19
21
|
exports.tokens = tokens;
|
|
22
|
+
exports.inputTokens = inputTokens;
|
|
23
|
+
exports.outputTokens = outputTokens;
|
|
24
|
+
exports.cacheTokens = cacheTokens;
|
|
20
25
|
const judge_js_1 = require("./judge.js");
|
|
26
|
+
const arg_match_js_1 = require("./arg-match.js");
|
|
21
27
|
/** Evaluate every check against a target. Pure — the shared core of `expect`
|
|
22
28
|
* (strict) and `measure` (scored). */
|
|
23
29
|
function evalChecks(target, checks) {
|
|
@@ -71,6 +77,63 @@ function tool(name) {
|
|
|
71
77
|
toJSON: () => ({ kind: "tool", name }),
|
|
72
78
|
};
|
|
73
79
|
}
|
|
80
|
+
/**
|
|
81
|
+
* The agent used tool `name` with at least one call whose `input` matches `args`
|
|
82
|
+
* — the **argument** half of the tool-call spy. Asserts not just *that* a tool was
|
|
83
|
+
* reached but *how* it was called (the image-request body carried the style suffix,
|
|
84
|
+
* the panel spawn requested a non-expert), which a completion grader can't see.
|
|
85
|
+
*
|
|
86
|
+
* Cross-reference: ≈ promptfoo `is-valid-function-call` / DeepEval `ToolCorrectnessMetric`.
|
|
87
|
+
*/
|
|
88
|
+
function toolWith(name, args) {
|
|
89
|
+
return {
|
|
90
|
+
kind: "toolWith",
|
|
91
|
+
eval: (t) => {
|
|
92
|
+
const calls = t.toolCalls.filter((c) => c.name === name);
|
|
93
|
+
const want = (0, arg_match_js_1.describeArgs)(args);
|
|
94
|
+
if (calls.length === 0) {
|
|
95
|
+
return no(`expected the agent to use tool "${name}" (with ${want}), but it used ${distinctToolNames(t.toolCalls)}`);
|
|
96
|
+
}
|
|
97
|
+
return calls.some((c) => (0, arg_match_js_1.matchesArgs)(c.input, args))
|
|
98
|
+
? ok(`agent used "${name}" with ${want}`)
|
|
99
|
+
: no(`agent used "${name}" but never with ${want} (saw ${calls
|
|
100
|
+
.map((c) => truncate((0, arg_match_js_1.stringifyValue)(c.input), 60))
|
|
101
|
+
.join("; ")})`);
|
|
102
|
+
},
|
|
103
|
+
toJSON: () => ({ kind: "toolWith", name, args: (0, arg_match_js_1.serializeArgs)(args) }),
|
|
104
|
+
};
|
|
105
|
+
}
|
|
106
|
+
/**
|
|
107
|
+
* The agent did NOT use tool `name` — the **safety / negative** assertion. With
|
|
108
|
+
* `args`, only calls whose `input` matches are forbidden (so "did not push to
|
|
109
|
+
* `main`" still allows pushing elsewhere; "no paid API call" forbids it outright).
|
|
110
|
+
* This is the highest-value, most-overlooked check, and the one a
|
|
111
|
+
* completion-grading eval structurally cannot make: it sees the agent's *decision*
|
|
112
|
+
* to act, not just its final text.
|
|
113
|
+
*
|
|
114
|
+
* Cross-reference: this negative/safety assertion of a *decision not to act* has
|
|
115
|
+
* no promptfoo / DeepEval / Inspect equivalent — they grade what the agent DID,
|
|
116
|
+
* not what it correctly refrained from doing.
|
|
117
|
+
*/
|
|
118
|
+
function notTool(name, args) {
|
|
119
|
+
return {
|
|
120
|
+
kind: "notTool",
|
|
121
|
+
eval: (t) => {
|
|
122
|
+
const calls = t.toolCalls.filter((c) => c.name === name);
|
|
123
|
+
const offending = args
|
|
124
|
+
? calls.filter((c) => (0, arg_match_js_1.matchesArgs)(c.input, args))
|
|
125
|
+
: calls;
|
|
126
|
+
const what = args ? `"${name}" with ${(0, arg_match_js_1.describeArgs)(args)}` : `"${name}"`;
|
|
127
|
+
const first = offending[0];
|
|
128
|
+
if (!first)
|
|
129
|
+
return ok(`agent never used ${what}`);
|
|
130
|
+
return no(`expected the agent NOT to use ${what}, but it did (${truncate((0, arg_match_js_1.stringifyValue)(first.input), 60)})`);
|
|
131
|
+
},
|
|
132
|
+
toJSON: () => args
|
|
133
|
+
? { kind: "notTool", name, args: (0, arg_match_js_1.serializeArgs)(args) }
|
|
134
|
+
: { kind: "notTool", name },
|
|
135
|
+
};
|
|
136
|
+
}
|
|
74
137
|
/** A skill resolved to this id (`<plugin>:<skill>`) without erroring. */
|
|
75
138
|
function skill(id) {
|
|
76
139
|
return {
|
|
@@ -315,4 +378,62 @@ function tokens(opts) {
|
|
|
315
378
|
toJSON: () => ({ kind: "tokens", max: opts.max }),
|
|
316
379
|
};
|
|
317
380
|
}
|
|
381
|
+
/**
|
|
382
|
+
* The run used at most `max` **fresh (uncached) input** tokens. The honest input
|
|
383
|
+
* side of a cost claim: a skill or CLAUDE.md injection adds input every turn, so
|
|
384
|
+
* a "compression" win on output can be erased here. (Cache reads are separate —
|
|
385
|
+
* see `cacheTokens`.)
|
|
386
|
+
*/
|
|
387
|
+
function inputTokens(opts) {
|
|
388
|
+
return {
|
|
389
|
+
kind: "inputTokens",
|
|
390
|
+
eval: (t) => {
|
|
391
|
+
const v = t.usage.inputTokens;
|
|
392
|
+
return v <= opts.max
|
|
393
|
+
? ok(`${String(v)} input tokens ≤ ${String(opts.max)}`)
|
|
394
|
+
: no(`expected ≤ ${String(opts.max)} input tokens, got ${String(v)}`);
|
|
395
|
+
},
|
|
396
|
+
toJSON: () => ({ kind: "inputTokens", max: opts.max }),
|
|
397
|
+
};
|
|
398
|
+
}
|
|
399
|
+
/** The run used at most `max` **output** tokens — the generation side. */
|
|
400
|
+
function outputTokens(opts) {
|
|
401
|
+
return {
|
|
402
|
+
kind: "outputTokens",
|
|
403
|
+
eval: (t) => {
|
|
404
|
+
const v = t.usage.outputTokens;
|
|
405
|
+
return v <= opts.max
|
|
406
|
+
? ok(`${String(v)} output tokens ≤ ${String(opts.max)}`)
|
|
407
|
+
: no(`expected ≤ ${String(opts.max)} output tokens, got ${String(v)}`);
|
|
408
|
+
},
|
|
409
|
+
toJSON: () => ({ kind: "outputTokens", max: opts.max }),
|
|
410
|
+
};
|
|
411
|
+
}
|
|
412
|
+
/**
|
|
413
|
+
* Bound the prompt-cache token classes a run uses. `maxCreation` caps tokens
|
|
414
|
+
* **written** to the cache (the ~1.25× write premium — a fresh/cold prompt);
|
|
415
|
+
* `maxRead` caps tokens **served** from cache (~0.1× input). Each constraint is
|
|
416
|
+
* checked only when provided; the check passes when every provided bound holds.
|
|
417
|
+
*/
|
|
418
|
+
function cacheTokens(opts) {
|
|
419
|
+
return {
|
|
420
|
+
kind: "cacheTokens",
|
|
421
|
+
eval: (t) => {
|
|
422
|
+
const created = t.usage.cacheCreationTokens;
|
|
423
|
+
const read = t.usage.cacheReadTokens;
|
|
424
|
+
if (opts.maxCreation !== undefined && created > opts.maxCreation) {
|
|
425
|
+
return no(`expected ≤ ${String(opts.maxCreation)} cache-creation tokens, got ${String(created)}`);
|
|
426
|
+
}
|
|
427
|
+
if (opts.maxRead !== undefined && read > opts.maxRead) {
|
|
428
|
+
return no(`expected ≤ ${String(opts.maxRead)} cache-read tokens, got ${String(read)}`);
|
|
429
|
+
}
|
|
430
|
+
return ok(`cache tokens within bounds (created ${String(created)}, read ${String(read)})`);
|
|
431
|
+
},
|
|
432
|
+
toJSON: () => ({
|
|
433
|
+
kind: "cacheTokens",
|
|
434
|
+
...(opts.maxCreation !== undefined && { maxCreation: opts.maxCreation }),
|
|
435
|
+
...(opts.maxRead !== undefined && { maxRead: opts.maxRead }),
|
|
436
|
+
}),
|
|
437
|
+
};
|
|
438
|
+
}
|
|
318
439
|
//# sourceMappingURL=check.js.map
|
package/dist/claude-code.d.ts
CHANGED
|
@@ -7,6 +7,7 @@
|
|
|
7
7
|
*/
|
|
8
8
|
export * from "./adapters/claude-code/plugin-loader.js";
|
|
9
9
|
export * from "./mock-model.js";
|
|
10
|
+
export { claudeCodeDriver, buildClaudeArgs, parseClaudeRun, claudeAvailable, } from "./harness-test.js";
|
|
10
11
|
export * from "./adapters/claude-code/dialect.js";
|
|
11
12
|
export * from "./adapters/claude-code/layout.js";
|
|
12
13
|
export * from "./adapters/claude-code/runtime.js";
|
package/dist/claude-code.js
CHANGED
|
@@ -14,6 +14,7 @@ var __exportStar = (this && this.__exportStar) || function(m, exports) {
|
|
|
14
14
|
for (var p in m) if (p !== "default" && !Object.prototype.hasOwnProperty.call(exports, p)) __createBinding(exports, m, p);
|
|
15
15
|
};
|
|
16
16
|
Object.defineProperty(exports, "__esModule", { value: true });
|
|
17
|
+
exports.claudeAvailable = exports.parseClaudeRun = exports.buildClaudeArgs = exports.claudeCodeDriver = void 0;
|
|
17
18
|
/**
|
|
18
19
|
* `vigiles/claude-code` — the Claude Code-specific harness pieces a *different*
|
|
19
20
|
* harness would swap out: the plugin/repo loader (reads real Claude Code plugin
|
|
@@ -23,6 +24,16 @@ Object.defineProperty(exports, "__esModule", { value: true });
|
|
|
23
24
|
*/
|
|
24
25
|
__exportStar(require("./adapters/claude-code/plugin-loader.js"), exports);
|
|
25
26
|
__exportStar(require("./mock-model.js"), exports);
|
|
27
|
+
// The Claude-Code harness-test transport — the default driver + its argv/parse
|
|
28
|
+
// helpers + the `claude` capability probe. Agnostic users never need these
|
|
29
|
+
// (`runHarnessTest` defaults to the CC driver), but they're exposed here — beside
|
|
30
|
+
// the Codex driver in `vigiles/codex` — for CC-specific tests/tooling. They are
|
|
31
|
+
// deliberately NOT on the agnostic `vigiles/testing` surface.
|
|
32
|
+
var harness_test_js_1 = require("./harness-test.js");
|
|
33
|
+
Object.defineProperty(exports, "claudeCodeDriver", { enumerable: true, get: function () { return harness_test_js_1.claudeCodeDriver; } });
|
|
34
|
+
Object.defineProperty(exports, "buildClaudeArgs", { enumerable: true, get: function () { return harness_test_js_1.buildClaudeArgs; } });
|
|
35
|
+
Object.defineProperty(exports, "parseClaudeRun", { enumerable: true, get: function () { return harness_test_js_1.parseClaudeRun; } });
|
|
36
|
+
Object.defineProperty(exports, "claudeAvailable", { enumerable: true, get: function () { return harness_test_js_1.claudeAvailable; } });
|
|
26
37
|
__exportStar(require("./adapters/claude-code/dialect.js"), exports);
|
|
27
38
|
__exportStar(require("./adapters/claude-code/layout.js"), exports);
|
|
28
39
|
__exportStar(require("./adapters/claude-code/runtime.js"), exports);
|