vigiles 4.1.0 → 5.0.1

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/README.md CHANGED
@@ -5,12 +5,11 @@
5
5
  <h1 align="center">vigiles</h1>
6
6
 
7
7
  <p align="center">
8
- <em>Quis custodiet ipsos custodes?</em> Who watches the watchmen?
8
+ <strong>Lint &amp; test the harness your AI agent runs on.</strong>
9
9
  </p>
10
10
 
11
11
  <p align="center">
12
- <strong>The missing linting + testing layer for agentic coding.</strong><br />
13
- vigiles <strong>lints</strong> the references your instruction files make — linter rules, file paths, scripts, code symbols — and <strong>tests</strong> whether your hooks, skills, and CLAUDE.md actually change what the agent does.
12
+ Your CLAUDE.md, hooks, and skills steer the agent — but nothing checks they're <em>true</em>, and nothing tests they <em>work</em>. vigiles does both.
14
13
  </p>
15
14
 
16
15
  <p align="center">
@@ -22,81 +21,17 @@
22
21
  ---
23
22
 
24
23
  `Agent = Model + Harness`. You'd never ship an app without a linter and a test
25
- suite — yet an AI agent steering your repo is trusted on vibes. vigiles is the
26
- deterministic layer for the harness: it **lints** the references your instruction
27
- files make and **tests** that your hooks and skills actually fire. Two independent
28
- pillars — adopt either, or both:
24
+ suite — yet the harness steering your agent runs on vibes. vigiles[^name] is the
25
+ deterministic layer for it, and does two independent things adopt either, or both:
29
26
 
30
- | | Pillar | What it does |
31
- | ----- | ------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
32
- | **①** | **Lint your instruction files** | Every linter rule, file path, script, and code symbol your CLAUDE.md cites is checked against reality, so stale references can't silently mislead the agent. → [guide](docs/verifying-instruction-files.md) |
33
- | **②** | **Test your harness** | Your hooks and skills are code vigiles tests they actually fire, **deterministically and free** (no model, no API key) before you pay for an eval. → [guide](docs/harness-testing.md) |
27
+ | | |
28
+ | ----------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
29
+ | **🔎 Lint** | Every file path, script, code symbol, and linter rule your CLAUDE.md cites is checked against reality so a renamed file or a disabled rule can't silently mislead the agent. **[→](docs/verifying-instruction-files.md)** |
30
+ | **🧪 Test** | Hooks, skills, and subagents are code. vigiles tests they _do their job_ and almost all of it is **deterministic, no API key**; the real-model evals run on your **Claude subscription**, not metered tokens. **[→](docs/harness-testing.md)** |
34
31
 
35
- Neither pillar depends on the other — pick the one that hurts today. **Works with
36
- Claude Code and Codex** ([`vigiles/codex`](docs/harnesses.md)) behind a five-port
37
- adapter; [custom adapters welcome](docs/authoring-an-adapter.md).
38
-
39
- ## ① Lint — your CLAUDE.md lies to your agent
40
-
41
- Your CLAUDE.md says _"enforce `eslint/no-console`."_ But it was switched off
42
- months ago — and the agent trusts the claim. (Same story for the file path it
43
- cites that got renamed, and the script that was deleted.)
44
-
45
- **Without vigiles:** nobody checks. The agent acts on fiction.
46
-
47
- **With vigiles:** `npx vigiles lint` resolves every reference against reality —
48
-
49
- ```text
50
- CLAUDE.md (inline mode):
51
- ✗ line 1: Rule "eslint/no-console" exists but is disabled in eslint config
52
- ✓ line 2: eslint/eqeqeq
53
- ✗ line 3: Rule "no-consoel" not found in eslint. Did you mean: "eslint/no-console"?
54
- ```
55
-
56
- It resolves rule names across **7 linter catalogs** — the rule exists **and is
57
- enabled** — and checks file paths, scripts, and code symbols the same way. Start
58
- with one comment, no new files:
59
-
60
- ```md
61
- <!-- vigiles:enforce eslint/no-console "Route output through logger.ts" -->
62
- ```
63
-
64
- Step up to a typed `.spec.ts` (compiled to CLAUDE.md, compiler-grade) when you
65
- want it. **[Full guide →](docs/verifying-instruction-files.md)**
66
-
67
- ## ② Test — does your harness actually fire?
68
-
69
- A hook can be wired wrong, a skill's description can fail to trigger, injected
70
- context can never reach the model — silently, all passing a naive "did it run?"
71
- check.
72
-
73
- **Without vigiles:** you assume your `--no-verify` guard blocks. You don't know.
74
-
75
- **With vigiles:** a deterministic test proves it — no model, no API key,
76
- milliseconds:
77
-
78
- ```typescript
79
- import { runHook } from "vigiles/testing";
80
-
81
- const r = runHook(guard, {
82
- hook_event_name: "PreToolUse",
83
- tool_name: "Bash",
84
- tool_input: { command: "git commit --no-verify" },
85
- });
86
- assert(r.blocked); // a red ✗ here means your hook silently lets it through
87
- ```
88
-
89
- ```text
90
- ✓ guard blocks --no-verify and allows a clean commit
91
-
92
- 2 passed.
93
- ```
94
-
95
- Three tiers, cheapest first: **`runHook`** (a hook's logic), **`runHarnessTest`**
96
- (the real agent CLI against a scripted mock model), **`runEval`** (the real model
97
- A/B with a significance gate). **Testing a skill?** `measureTriggerRate` checks
98
- its description actually **fires** across varied prompts (recall) without
99
- hijacking unrelated ones (precision). **[Full guide →](docs/harness-testing.md)**
32
+ Pick the one that hurts today. **Works with Claude Code and Codex**
33
+ ([`vigiles/codex`](docs/harnesses.md)) behind a five-port adapter;
34
+ [custom adapters welcome](docs/authoring-an-adapter.md).
100
35
 
101
36
  ## Quick start
102
37
 
@@ -105,7 +40,7 @@ hijacking unrelated ones (precision). **[Full guide →](docs/harness-testing.md
105
40
  ```text
106
41
  Install vigiles in this repo and run it. Verify my CLAUDE.md / AGENTS.md
107
42
  references and show me what's stale, then write and run a harness test for one
108
- of my hooks or skills. Use good defaults (both pillars, non-interactive), but
43
+ of my hooks or skills. Use good defaults (lint + test, non-interactive), but
109
44
  ask me first whether to gate it in CI, whether to add a real-model eval, and
110
45
  whether to enforce strictly (--strict).
111
46
  ```
@@ -113,7 +48,7 @@ whether to enforce strictly (--strict).
113
48
  Or do it yourself:
114
49
 
115
50
  ```bash
116
- npx vigiles init # sets up BOTH pillars: spec + harness test + CI + plugin
51
+ npx vigiles init # sets up lint + test: spec + harness test + CI + plugin
117
52
  ```
118
53
 
119
54
  It's interactive in a terminal and non-interactive for agents/CI (or with
@@ -124,7 +59,7 @@ tell your agent _"test my skills"_ and it picks the tier and writes the test.
124
59
  <details>
125
60
  <summary>What <code>init</code> sets up</summary>
126
61
 
127
- - **Both pillars** by default; scope with `--lint` / `--test` (one or both).
62
+ - **Both lint and test** by default; scope with `--lint` / `--test` (one or both).
128
63
  - Adds `vigiles` to your `devDependencies`.
129
64
  - Installs the Claude Code plugin (skills + hooks) via the marketplace —
130
65
  globally, never vendored into your repo.
@@ -140,6 +75,61 @@ Prefer to write tests yourself? They can be JS **or** TS
140
75
 
141
76
  </details>
142
77
 
78
+ ## ① Lint — your CLAUDE.md lies to your agent
79
+
80
+ Your CLAUDE.md points the agent at `src/auth/login.ts` and tells it to run
81
+ `npm run check`. But the file moved to `src/auth/session.ts` six commits ago, and
82
+ the script was renamed. The agent trusts the stale claim and acts on fiction.
83
+
84
+ `npx vigiles lint` resolves every reference against reality:
85
+
86
+ ```text
87
+ CLAUDE.md:
88
+ ✗ src/auth/login.ts — no such file (renamed or moved?)
89
+ ✗ npm run check — not in package.json. Did you mean: "check:types"?
90
+ ✓ @typescript-eslint/no-floating-promises — exists and enabled in eslint config
91
+ ```
92
+
93
+ File paths, scripts, and code symbols — plus linter rules across **7 catalogs**
94
+ (the rule exists **and is enabled**). Start with one inline comment, no new files;
95
+ step up to a typed `.spec.ts` (compiled to CLAUDE.md, compiler-grade) when you want
96
+ it. **[Full guide →](docs/verifying-instruction-files.md)**
97
+
98
+ ## ② Test — does your harness do its job?
99
+
100
+ A hook can be wired wrong. A skill's description can fail to trigger — or hijack
101
+ unrelated prompts. Injected context can never reach the model. All of it passes a
102
+ naive "did it run?" check. vigiles tests the assembled harness for real:
103
+
104
+ ```typescript
105
+ import { runHook } from "vigiles/testing";
106
+
107
+ const r = runHook(guard, {
108
+ hook_event_name: "PreToolUse",
109
+ tool_name: "Bash",
110
+ tool_input: { command: "git commit --no-verify" },
111
+ });
112
+ assert(r.blocked); // a red ✗ means your guard silently lets it through
113
+ ```
114
+
115
+ It goes well past _"did it fire?"_:
116
+
117
+ - **Hooks block** what they must — `runHook`, or the real agent CLI via `runHarnessTest`.
118
+ - **Skills trigger** on the right prompts and stay quiet on the wrong ones — recall _and_ precision (`measureTriggerRate`).
119
+ - **Behaviour is good** — score a skill's output directly, or A/B it on-vs-off for the real lift over no-skill (`measure` / `runEval`, with significance testing).
120
+ - **Safety holds** — the agent _didn't_ push to the wrong branch or hit a paid API; `interceptTools` catches the attempt so the side effect never happens.
121
+
122
+ **The eval you can actually afford.** Almost every tier runs with **no model and
123
+ no API key** — milliseconds, on every commit. The rest drive your own `claude` CLI:
124
+
125
+ | | Runs on | Cost |
126
+ | ---------------------- | ----------------------- | ------------------------------------------- |
127
+ | promptfoo, DeepEval, … | metered API SDK | billed **per token, every run** |
128
+ | **vigiles** | your Claude Pro/Max sub | **$0 extra** — and most tiers need no model |
129
+
130
+ That's why you can eval your harness on every change, not just once.
131
+ **[How it works →](docs/harness-testing.md)** · **[Why it's affordable →](docs/eval-architecture.md)** · **[Safety model →](docs/safety.md)**
132
+
143
133
  ## More
144
134
 
145
135
  - **[CLI & GitHub Action →](docs/cli.md)** — every command, the Action (inputs / output / versioning), the Claude Code plugin, and the five `lint` rules.
@@ -150,3 +140,5 @@ Prefer to write tests yourself? They can be JS **or** TS
150
140
  ## License
151
141
 
152
142
  [MIT](LICENSE)
143
+
144
+ [^name]: **vigiles** — the watchmen of ancient Rome, who guarded the city (and fought its fires) by night. _Quis custodiet ipsos custodes?_ — "who watches the watchmen?" (Juvenal, _Satire VI_).
@@ -3,7 +3,7 @@
3
3
  *
4
4
  * A subagent declares an allowed-tools contract in its frontmatter (`tools:`).
5
5
  * But that field is documentation, not a hard runtime boundary (Claude Code
6
- * issue #54898): permissions are session-wide, a subagent inherits the parent
6
+ * #4740/#21460, SDK #172): permissions are session-wide, a subagent inherits the parent
7
7
  * session's grants, and `tools:` only filters what's *offered* — it can't deny
8
8
  * what the session allows. The deterministic layer that actually closes the gap
9
9
  * is a **PreToolUse hook** that blocks any tool the active agent's contract
@@ -4,7 +4,7 @@
4
4
  *
5
5
  * A subagent declares an allowed-tools contract in its frontmatter (`tools:`).
6
6
  * But that field is documentation, not a hard runtime boundary (Claude Code
7
- * issue #54898): permissions are session-wide, a subagent inherits the parent
7
+ * #4740/#21460, SDK #172): permissions are session-wide, a subagent inherits the parent
8
8
  * session's grants, and `tools:` only filters what's *offered* — it can't deny
9
9
  * what the session allows. The deterministic layer that actually closes the gap
10
10
  * is a **PreToolUse hook** that blocks any tool the active agent's contract
@@ -0,0 +1,28 @@
1
+ /**
2
+ * `ArgMatcher` — a small, serializable matcher over a tool call's `input`.
3
+ *
4
+ * Shared by the check vocabulary (`toolWith` / `notTool` in `src/check.ts`) and
5
+ * the tool-interception seam (`src/tool-intercept.ts`), so "did the agent call
6
+ * this tool with these arguments?" means the same thing whether you're *asserting*
7
+ * on a captured call or *intercepting* one before it runs. Pure + model-free.
8
+ */
9
+ /**
10
+ * A declarative matcher over a tool call's `input`, keyed by **dot-path** (e.g.
11
+ * `"body.prompt"`). Each value is matched against the value at that path: a
12
+ * `RegExp` is a pattern over the stringified value (use this for "contains"), and
13
+ * a `string`/`number`/`boolean` is an **exact** match (use this for "equals", e.g.
14
+ * a push target). All keys must match (AND). Serializable, so anything carrying
15
+ * one still round-trips through `toJSON`.
16
+ */
17
+ export type ArgMatcher = Record<string, string | number | boolean | RegExp>;
18
+ /** Render any tool-input value as a string for matching / messages. */
19
+ export declare function stringifyValue(v: unknown): string;
20
+ /** Resolve a dot-path (`"a.b.c"`) within an arbitrary value, or undefined. */
21
+ export declare function getPath(obj: unknown, path: string): unknown;
22
+ /** Does `input` satisfy every entry of `matcher`? (RegExp = pattern, else exact.) */
23
+ export declare function matchesArgs(input: unknown, matcher: ArgMatcher): boolean;
24
+ /** A human-readable form of a matcher for failure messages. */
25
+ export declare function describeArgs(matcher: ArgMatcher): string;
26
+ /** Serialize a matcher for `toJSON` (RegExp → its string form). */
27
+ export declare function serializeArgs(matcher: ArgMatcher): Record<string, string | number | boolean>;
28
+ //# sourceMappingURL=arg-match.d.ts.map
@@ -0,0 +1,61 @@
1
+ "use strict";
2
+ /**
3
+ * `ArgMatcher` — a small, serializable matcher over a tool call's `input`.
4
+ *
5
+ * Shared by the check vocabulary (`toolWith` / `notTool` in `src/check.ts`) and
6
+ * the tool-interception seam (`src/tool-intercept.ts`), so "did the agent call
7
+ * this tool with these arguments?" means the same thing whether you're *asserting*
8
+ * on a captured call or *intercepting* one before it runs. Pure + model-free.
9
+ */
10
+ Object.defineProperty(exports, "__esModule", { value: true });
11
+ exports.stringifyValue = stringifyValue;
12
+ exports.getPath = getPath;
13
+ exports.matchesArgs = matchesArgs;
14
+ exports.describeArgs = describeArgs;
15
+ exports.serializeArgs = serializeArgs;
16
+ /** Render any tool-input value as a string for matching / messages. */
17
+ function stringifyValue(v) {
18
+ if (typeof v === "string")
19
+ return v;
20
+ if (typeof v === "object" && v !== null) {
21
+ try {
22
+ return JSON.stringify(v) ?? "[object]";
23
+ }
24
+ catch {
25
+ return "[object]";
26
+ }
27
+ }
28
+ return String(v);
29
+ }
30
+ /** Resolve a dot-path (`"a.b.c"`) within an arbitrary value, or undefined. */
31
+ function getPath(obj, path) {
32
+ let cur = obj;
33
+ for (const part of path.split(".")) {
34
+ if (cur === null || typeof cur !== "object")
35
+ return undefined;
36
+ cur = cur[part];
37
+ }
38
+ return cur;
39
+ }
40
+ /** Does `input` satisfy every entry of `matcher`? (RegExp = pattern, else exact.) */
41
+ function matchesArgs(input, matcher) {
42
+ return Object.entries(matcher).every(([key, m]) => {
43
+ const value = getPath(input, key);
44
+ return m instanceof RegExp ? m.test(stringifyValue(value)) : value === m;
45
+ });
46
+ }
47
+ /** A human-readable form of a matcher for failure messages. */
48
+ function describeArgs(matcher) {
49
+ return Object.entries(matcher)
50
+ .map(([k, m]) => `${k}=${m instanceof RegExp ? String(m) : JSON.stringify(m)}`)
51
+ .join(", ");
52
+ }
53
+ /** Serialize a matcher for `toJSON` (RegExp → its string form). */
54
+ function serializeArgs(matcher) {
55
+ const out = {};
56
+ for (const [k, m] of Object.entries(matcher)) {
57
+ out[k] = m instanceof RegExp ? String(m) : m;
58
+ }
59
+ return out;
60
+ }
61
+ //# sourceMappingURL=arg-match.js.map
package/dist/check.d.ts CHANGED
@@ -18,6 +18,8 @@
18
18
  */
19
19
  import type { Trace } from "./harness-test.js";
20
20
  import type { HookRunResult } from "./run-hook.js";
21
+ import { type ArgMatcher } from "./arg-match.js";
22
+ export type { ArgMatcher };
21
23
  /** The outcome of evaluating one check against one result. */
22
24
  export interface CheckResult {
23
25
  /** Did the check hold? */
@@ -54,6 +56,28 @@ export declare function evalChecks<T>(target: T, checks: readonly Check<T>[]): C
54
56
  export declare function assertChecks<T>(target: T, checks: readonly Check<T>[]): void;
55
57
  /** The agent invoked a tool by this name (regardless of result). */
56
58
  export declare function tool(name: string): Check<Trace>;
59
+ /**
60
+ * The agent used tool `name` with at least one call whose `input` matches `args`
61
+ * — the **argument** half of the tool-call spy. Asserts not just *that* a tool was
62
+ * reached but *how* it was called (the image-request body carried the style suffix,
63
+ * the panel spawn requested a non-expert), which a completion grader can't see.
64
+ *
65
+ * Cross-reference: ≈ promptfoo `is-valid-function-call` / DeepEval `ToolCorrectnessMetric`.
66
+ */
67
+ export declare function toolWith(name: string, args: ArgMatcher): Check<Trace>;
68
+ /**
69
+ * The agent did NOT use tool `name` — the **safety / negative** assertion. With
70
+ * `args`, only calls whose `input` matches are forbidden (so "did not push to
71
+ * `main`" still allows pushing elsewhere; "no paid API call" forbids it outright).
72
+ * This is the highest-value, most-overlooked check, and the one a
73
+ * completion-grading eval structurally cannot make: it sees the agent's *decision*
74
+ * to act, not just its final text.
75
+ *
76
+ * Cross-reference: this negative/safety assertion of a *decision not to act* has
77
+ * no promptfoo / DeepEval / Inspect equivalent — they grade what the agent DID,
78
+ * not what it correctly refrained from doing.
79
+ */
80
+ export declare function notTool(name: string, args?: ArgMatcher): Check<Trace>;
57
81
  /** A skill resolved to this id (`<plugin>:<skill>`) without erroring. */
58
82
  export declare function skill(id: string): Check<Trace>;
59
83
  /** The agent's final output contains a substring / matches a RegExp. */
@@ -112,8 +136,13 @@ interface UsageTrace {
112
136
  readonly usage: {
113
137
  readonly costUsd: number;
114
138
  readonly durationMs: number;
139
+ /** Fresh (uncached) input tokens, billed at full input price. */
115
140
  readonly inputTokens: number;
116
141
  readonly outputTokens: number;
142
+ /** Tokens written to the prompt cache this run (~1.25× input price). */
143
+ readonly cacheCreationTokens: number;
144
+ /** Tokens served from the prompt cache this run (~0.1× input price). */
145
+ readonly cacheReadTokens: number;
117
146
  };
118
147
  }
119
148
  /** The run cost at most `maxUsd`. */
@@ -128,5 +157,27 @@ export declare function latency(opts: {
128
157
  export declare function tokens(opts: {
129
158
  max: number;
130
159
  }): Check<UsageTrace>;
131
- export {};
160
+ /**
161
+ * The run used at most `max` **fresh (uncached) input** tokens. The honest input
162
+ * side of a cost claim: a skill or CLAUDE.md injection adds input every turn, so
163
+ * a "compression" win on output can be erased here. (Cache reads are separate —
164
+ * see `cacheTokens`.)
165
+ */
166
+ export declare function inputTokens(opts: {
167
+ max: number;
168
+ }): Check<UsageTrace>;
169
+ /** The run used at most `max` **output** tokens — the generation side. */
170
+ export declare function outputTokens(opts: {
171
+ max: number;
172
+ }): Check<UsageTrace>;
173
+ /**
174
+ * Bound the prompt-cache token classes a run uses. `maxCreation` caps tokens
175
+ * **written** to the cache (the ~1.25× write premium — a fresh/cold prompt);
176
+ * `maxRead` caps tokens **served** from cache (~0.1× input). Each constraint is
177
+ * checked only when provided; the check passes when every provided bound holds.
178
+ */
179
+ export declare function cacheTokens(opts: {
180
+ maxCreation?: number;
181
+ maxRead?: number;
182
+ }): Check<UsageTrace>;
132
183
  //# sourceMappingURL=check.d.ts.map
package/dist/check.js CHANGED
@@ -3,6 +3,8 @@ Object.defineProperty(exports, "__esModule", { value: true });
3
3
  exports.evalChecks = evalChecks;
4
4
  exports.assertChecks = assertChecks;
5
5
  exports.tool = tool;
6
+ exports.toolWith = toolWith;
7
+ exports.notTool = notTool;
6
8
  exports.skill = skill;
7
9
  exports.output = output;
8
10
  exports.hookFired = hookFired;
@@ -17,7 +19,11 @@ exports.judged = judged;
17
19
  exports.cost = cost;
18
20
  exports.latency = latency;
19
21
  exports.tokens = tokens;
22
+ exports.inputTokens = inputTokens;
23
+ exports.outputTokens = outputTokens;
24
+ exports.cacheTokens = cacheTokens;
20
25
  const judge_js_1 = require("./judge.js");
26
+ const arg_match_js_1 = require("./arg-match.js");
21
27
  /** Evaluate every check against a target. Pure — the shared core of `expect`
22
28
  * (strict) and `measure` (scored). */
23
29
  function evalChecks(target, checks) {
@@ -71,6 +77,63 @@ function tool(name) {
71
77
  toJSON: () => ({ kind: "tool", name }),
72
78
  };
73
79
  }
80
+ /**
81
+ * The agent used tool `name` with at least one call whose `input` matches `args`
82
+ * — the **argument** half of the tool-call spy. Asserts not just *that* a tool was
83
+ * reached but *how* it was called (the image-request body carried the style suffix,
84
+ * the panel spawn requested a non-expert), which a completion grader can't see.
85
+ *
86
+ * Cross-reference: ≈ promptfoo `is-valid-function-call` / DeepEval `ToolCorrectnessMetric`.
87
+ */
88
+ function toolWith(name, args) {
89
+ return {
90
+ kind: "toolWith",
91
+ eval: (t) => {
92
+ const calls = t.toolCalls.filter((c) => c.name === name);
93
+ const want = (0, arg_match_js_1.describeArgs)(args);
94
+ if (calls.length === 0) {
95
+ return no(`expected the agent to use tool "${name}" (with ${want}), but it used ${distinctToolNames(t.toolCalls)}`);
96
+ }
97
+ return calls.some((c) => (0, arg_match_js_1.matchesArgs)(c.input, args))
98
+ ? ok(`agent used "${name}" with ${want}`)
99
+ : no(`agent used "${name}" but never with ${want} (saw ${calls
100
+ .map((c) => truncate((0, arg_match_js_1.stringifyValue)(c.input), 60))
101
+ .join("; ")})`);
102
+ },
103
+ toJSON: () => ({ kind: "toolWith", name, args: (0, arg_match_js_1.serializeArgs)(args) }),
104
+ };
105
+ }
106
+ /**
107
+ * The agent did NOT use tool `name` — the **safety / negative** assertion. With
108
+ * `args`, only calls whose `input` matches are forbidden (so "did not push to
109
+ * `main`" still allows pushing elsewhere; "no paid API call" forbids it outright).
110
+ * This is the highest-value, most-overlooked check, and the one a
111
+ * completion-grading eval structurally cannot make: it sees the agent's *decision*
112
+ * to act, not just its final text.
113
+ *
114
+ * Cross-reference: this negative/safety assertion of a *decision not to act* has
115
+ * no promptfoo / DeepEval / Inspect equivalent — they grade what the agent DID,
116
+ * not what it correctly refrained from doing.
117
+ */
118
+ function notTool(name, args) {
119
+ return {
120
+ kind: "notTool",
121
+ eval: (t) => {
122
+ const calls = t.toolCalls.filter((c) => c.name === name);
123
+ const offending = args
124
+ ? calls.filter((c) => (0, arg_match_js_1.matchesArgs)(c.input, args))
125
+ : calls;
126
+ const what = args ? `"${name}" with ${(0, arg_match_js_1.describeArgs)(args)}` : `"${name}"`;
127
+ const first = offending[0];
128
+ if (!first)
129
+ return ok(`agent never used ${what}`);
130
+ return no(`expected the agent NOT to use ${what}, but it did (${truncate((0, arg_match_js_1.stringifyValue)(first.input), 60)})`);
131
+ },
132
+ toJSON: () => args
133
+ ? { kind: "notTool", name, args: (0, arg_match_js_1.serializeArgs)(args) }
134
+ : { kind: "notTool", name },
135
+ };
136
+ }
74
137
  /** A skill resolved to this id (`<plugin>:<skill>`) without erroring. */
75
138
  function skill(id) {
76
139
  return {
@@ -315,4 +378,62 @@ function tokens(opts) {
315
378
  toJSON: () => ({ kind: "tokens", max: opts.max }),
316
379
  };
317
380
  }
381
+ /**
382
+ * The run used at most `max` **fresh (uncached) input** tokens. The honest input
383
+ * side of a cost claim: a skill or CLAUDE.md injection adds input every turn, so
384
+ * a "compression" win on output can be erased here. (Cache reads are separate —
385
+ * see `cacheTokens`.)
386
+ */
387
+ function inputTokens(opts) {
388
+ return {
389
+ kind: "inputTokens",
390
+ eval: (t) => {
391
+ const v = t.usage.inputTokens;
392
+ return v <= opts.max
393
+ ? ok(`${String(v)} input tokens ≤ ${String(opts.max)}`)
394
+ : no(`expected ≤ ${String(opts.max)} input tokens, got ${String(v)}`);
395
+ },
396
+ toJSON: () => ({ kind: "inputTokens", max: opts.max }),
397
+ };
398
+ }
399
+ /** The run used at most `max` **output** tokens — the generation side. */
400
+ function outputTokens(opts) {
401
+ return {
402
+ kind: "outputTokens",
403
+ eval: (t) => {
404
+ const v = t.usage.outputTokens;
405
+ return v <= opts.max
406
+ ? ok(`${String(v)} output tokens ≤ ${String(opts.max)}`)
407
+ : no(`expected ≤ ${String(opts.max)} output tokens, got ${String(v)}`);
408
+ },
409
+ toJSON: () => ({ kind: "outputTokens", max: opts.max }),
410
+ };
411
+ }
412
+ /**
413
+ * Bound the prompt-cache token classes a run uses. `maxCreation` caps tokens
414
+ * **written** to the cache (the ~1.25× write premium — a fresh/cold prompt);
415
+ * `maxRead` caps tokens **served** from cache (~0.1× input). Each constraint is
416
+ * checked only when provided; the check passes when every provided bound holds.
417
+ */
418
+ function cacheTokens(opts) {
419
+ return {
420
+ kind: "cacheTokens",
421
+ eval: (t) => {
422
+ const created = t.usage.cacheCreationTokens;
423
+ const read = t.usage.cacheReadTokens;
424
+ if (opts.maxCreation !== undefined && created > opts.maxCreation) {
425
+ return no(`expected ≤ ${String(opts.maxCreation)} cache-creation tokens, got ${String(created)}`);
426
+ }
427
+ if (opts.maxRead !== undefined && read > opts.maxRead) {
428
+ return no(`expected ≤ ${String(opts.maxRead)} cache-read tokens, got ${String(read)}`);
429
+ }
430
+ return ok(`cache tokens within bounds (created ${String(created)}, read ${String(read)})`);
431
+ },
432
+ toJSON: () => ({
433
+ kind: "cacheTokens",
434
+ ...(opts.maxCreation !== undefined && { maxCreation: opts.maxCreation }),
435
+ ...(opts.maxRead !== undefined && { maxRead: opts.maxRead }),
436
+ }),
437
+ };
438
+ }
318
439
  //# sourceMappingURL=check.js.map
@@ -7,6 +7,7 @@
7
7
  */
8
8
  export * from "./adapters/claude-code/plugin-loader.js";
9
9
  export * from "./mock-model.js";
10
+ export { claudeCodeDriver, buildClaudeArgs, parseClaudeRun, claudeAvailable, } from "./harness-test.js";
10
11
  export * from "./adapters/claude-code/dialect.js";
11
12
  export * from "./adapters/claude-code/layout.js";
12
13
  export * from "./adapters/claude-code/runtime.js";
@@ -14,6 +14,7 @@ var __exportStar = (this && this.__exportStar) || function(m, exports) {
14
14
  for (var p in m) if (p !== "default" && !Object.prototype.hasOwnProperty.call(exports, p)) __createBinding(exports, m, p);
15
15
  };
16
16
  Object.defineProperty(exports, "__esModule", { value: true });
17
+ exports.claudeAvailable = exports.parseClaudeRun = exports.buildClaudeArgs = exports.claudeCodeDriver = void 0;
17
18
  /**
18
19
  * `vigiles/claude-code` — the Claude Code-specific harness pieces a *different*
19
20
  * harness would swap out: the plugin/repo loader (reads real Claude Code plugin
@@ -23,6 +24,16 @@ Object.defineProperty(exports, "__esModule", { value: true });
23
24
  */
24
25
  __exportStar(require("./adapters/claude-code/plugin-loader.js"), exports);
25
26
  __exportStar(require("./mock-model.js"), exports);
27
+ // The Claude-Code harness-test transport — the default driver + its argv/parse
28
+ // helpers + the `claude` capability probe. Agnostic users never need these
29
+ // (`runHarnessTest` defaults to the CC driver), but they're exposed here — beside
30
+ // the Codex driver in `vigiles/codex` — for CC-specific tests/tooling. They are
31
+ // deliberately NOT on the agnostic `vigiles/testing` surface.
32
+ var harness_test_js_1 = require("./harness-test.js");
33
+ Object.defineProperty(exports, "claudeCodeDriver", { enumerable: true, get: function () { return harness_test_js_1.claudeCodeDriver; } });
34
+ Object.defineProperty(exports, "buildClaudeArgs", { enumerable: true, get: function () { return harness_test_js_1.buildClaudeArgs; } });
35
+ Object.defineProperty(exports, "parseClaudeRun", { enumerable: true, get: function () { return harness_test_js_1.parseClaudeRun; } });
36
+ Object.defineProperty(exports, "claudeAvailable", { enumerable: true, get: function () { return harness_test_js_1.claudeAvailable; } });
26
37
  __exportStar(require("./adapters/claude-code/dialect.js"), exports);
27
38
  __exportStar(require("./adapters/claude-code/layout.js"), exports);
28
39
  __exportStar(require("./adapters/claude-code/runtime.js"), exports);