vigiles 4.0.2 → 5.0.0

package/README.md

@@ -27,10 +27,10 @@ deterministic layer for the harness: it **lints** the references your instructio
 files make and **tests** that your hooks and skills actually fire. Two independent
 pillars — adopt either, or both:
 
-|       | Pillar                          | What it does                                                                                                                                                                                                |
-| ----- | ------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
-| **①** | **Lint your instruction files** | Every linter rule, file path, script, and code symbol your CLAUDE.md cites is checked against reality, so stale references can't silently mislead the agent. → [guide](docs/verifying-instruction-files.md) |
-| **②** | **Test your harness**           | Your hooks and skills are code — vigiles tests they actually fire, **deterministically and free** (no model, no API key) before you pay for an eval. → [guide](docs/harness-testing.md)                     |
+|       | Pillar                          | What it does                                                                                                                                                                                                                                                              |
+| ----- | ------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| **①** | **Lint your instruction files** | Every linter rule, file path, script, and code symbol your CLAUDE.md cites is checked against reality, so stale references can't silently mislead the agent. → [guide](docs/verifying-instruction-files.md)                                                               |
+| **②** | **Test your harness**           | Your hooks and skills are code — vigiles tests they actually fire, **deterministically and free** (no model, no API key); and when a question _does_ need a real-model eval, it runs on your **Claude subscription**, not metered API. → [guide](docs/harness-testing.md) |
 
 Neither pillar depends on the other — pick the one that hurts today. **Works with
 Claude Code and Codex** ([`vigiles/codex`](docs/harnesses.md)) behind a five-port
@@ -93,10 +93,42 @@ assert(r.blocked); // a red ✗ here means your hook silently lets it through
 ```
 
 Three tiers, cheapest first: **`runHook`** (a hook's logic), **`runHarnessTest`**
-(the real agent CLI against a scripted mock model), **`runEval`** (the real model
-A/B with a significance gate). **Testing a skill?** `measureTriggerRate` checks
-its description actually **fires** across varied prompts (recall) without
-hijacking unrelated ones (precision). **[Full guide →](docs/harness-testing.md)**
+(the real agent CLI against a scripted mock model), and the real-model scored tier
+(**`measure`** / **`runEval`**). **Testing a skill?** Two questions, both covered:
+does its description **fire** (`measureTriggerRate` — recall across varied prompts
+without hijacking unrelated ones, precision), **and** does its guidance actually
+**work**. For "is this exact skill any good?" score the output directly —
+`measure({ checks: [judged(rubric)] })` + `assertRates` (the **absolute** oracle,
+what promptfoo/DeepEval lead with; no on/off baseline needed). When you need the
+**relative** lift over no-skill — regression, or proving the change isn't noise —
+A/B it on-vs-off with `runEval` + `assertSignificant`. Description _and_ behavior,
+not just one. **Need a safety property** — that the agent
+**didn't** push to the wrong branch or call a paid API? `notTool` + `interceptTools`
+intercept the tool in the real hook layer, so the attempt is caught and the side
+effect never happens. **[Full guide →](docs/harness-testing.md)** · vigiles runs
+foreign code (and a real model) safely by default — **[safety model →](docs/safety.md)**
+
+Most of what real plugins do is testable cheaply — fire / trigger / contract /
+safety, plus **record-replay** for the tool/API results a skill consumes (recorded
+once from the real tool, replayed deterministically — no live service, no Docker).
+That covers ~90%+ of real plugin surface on your subscription; the rare case that
+needs a real browser or database **composes with Docker** rather than us
+reinventing the sandbox. **[What we test, how →](research/eval-coverage-and-isolation.md)**
+
+**Affordable by design — the eval you can actually run.** Almost nobody evals
+their harness, because the usual tools (promptfoo, DeepEval, …) hit the API SDK
+and bill **per token on every run**. vigiles inverts that: most questions are
+answered with **no model at all** (free, every commit), and when you do reach for
+a real-model eval, vigiles drives your `claude` CLI — so it runs on the **Pro/Max
+subscription you already pay for**, not metered API billing. CI runs only the free
+deterministic tiers; you run the real-model eval where the subscription already is
+— a Claude Code session or locally — when it's worth it, not on every PR.
+
+This affordability story is **ToS-clean**: vigiles drives _your own_ `claude` CLI
+to test _your own_ harness on _your own_ subscription — the same thing you do when
+you run Claude Code. (The Claude Agent SDK's ToS restricts _productizing_ claude.ai
+login/limits in a third-party offering; running your own tests on your own sub is
+exactly the supported posture, not that.)
 
 ## Quick start
 

package/dist/adapter-registry.d.ts

@@ -31,12 +31,56 @@ export interface DetectResult {
 export declare function detectAdapterResult(root: string): DetectResult;
 /** The detected adapter (highest specificity), else the default (Claude Code). */
 export declare function detectAdapter(root: string): HarnessAdapter;
-/** Look up a registered adapter by `name` (e.g. for a `--harness` override). */
+/** Lower-case, trim, and map a short alias to its canonical adapter name. */
+export declare function normalizeHarnessName(name: string): string;
+/** Look up a registered adapter by `name` (alias-aware, e.g. `claude`). */
 export declare function getAdapter(name: string): HarnessAdapter | undefined;
+/**
+ * The adapter whose instruction file is `filename` (e.g. `AGENTS.md` → codex,
+ * `CLAUDE.md` → claude-code), if any. The per-spec disambiguation signal: a
+ * `<file>.spec.ts` compiles a `<file>` instruction file, so the filename names
+ * the harness more specifically than config/detect for THAT spec.
+ */
+export declare function adapterForInstructionFile(filename: string): HarnessAdapter | undefined;
 /**
  * Resolve the adapter for a command: an explicit `--harness <name>` wins (throws
  * if unknown); otherwise auto-detect from `root`. The single entry point the CLI
  * uses so detection + override live in one place.
  */
 export declare function resolveAdapter(root: string, harness?: string): HarnessAdapter;
+/** Normalize a config `harness` value (string | string[]) to a canonical list. */
+export declare function normalizeHarnessList(harness?: string | readonly string[]): string[];
+/**
+ * The adapter chosen for a single-dialect operation. A discriminated union so an
+ * invalid state — a "notice" with no message, or a clean pick carrying a stray
+ * string — is unrepresentable: `kind: "ok"` has no `notice`, `kind: "notice"`
+ * always carries a non-empty one. Both variants carry the `adapter`.
+ */
+export type HarnessSelection = {
+    readonly kind: "ok";
+    readonly adapter: HarnessAdapter;
+} | {
+    readonly kind: "notice";
+    readonly adapter: HarnessAdapter;
+    readonly notice: string;
+};
+/**
+ * Resolve the single harness a compile/lint operation should use, with explicit
+ * precedence — the deterministic replacement for sniffing the cwd:
+ *
+ *   1. `--harness=` flag (wins; throws if unknown).
+ *   2. config `harness` resolving to a single entry → use it.
+ *   3. config `harness` with multiple entries → use the first, with a loud notice.
+ *   4. no config → auto-detect, with a loud notice when the repo is ambiguous.
+ *
+ * `configHarness` is parsed once (alias-normalized) at the call site and passed
+ * in; this function re-normalizes idempotently so it's safe either way. Pure
+ * (besides reading `root`'s layout for detection) so the precedence is
+ * unit-testable without a real compile. See research/multi-harness-compile.md.
+ */
+export declare function resolveHarnessSelection(opts: {
+    root: string;
+    flag?: string;
+    configHarness?: string | readonly string[];
+}): HarnessSelection;
 //# sourceMappingURL=adapter-registry.d.ts.map

package/dist/adapter-registry.js

@@ -3,8 +3,12 @@ Object.defineProperty(exports, "__esModule", { value: true });
 exports.ADAPTERS = exports.defaultAdapter = void 0;
 exports.detectAdapterResult = detectAdapterResult;
 exports.detectAdapter = detectAdapter;
+exports.normalizeHarnessName = normalizeHarnessName;
 exports.getAdapter = getAdapter;
+exports.adapterForInstructionFile = adapterForInstructionFile;
 exports.resolveAdapter = resolveAdapter;
+exports.normalizeHarnessList = normalizeHarnessList;
+exports.resolveHarnessSelection = resolveHarnessSelection;
 const adapter_js_1 = require("./adapters/claude-code/adapter.js");
 const adapter_js_2 = require("./adapters/codex/adapter.js");
 /** The default adapter when detection finds no harness markers. */
@@ -32,9 +36,32 @@ function detectAdapterResult(root) {
 function detectAdapter(root) {
     return detectAdapterResult(root).adapter;
 }
-/** Look up a registered adapter by `name` (e.g. for a `--harness` override). */
+/**
+ * Short-name aliases accepted anywhere a harness name is supplied (config,
+ * `--harness=`). `init` historically uses `"claude"`; the canonical adapter name
+ * is `"claude-code"`. Normalizing here keeps selection and the registry in sync.
+ */
+const HARNESS_ALIASES = {
+    claude: "claude-code",
+};
+/** Lower-case, trim, and map a short alias to its canonical adapter name. */
+function normalizeHarnessName(name) {
+    const n = name.trim().toLowerCase();
+    return HARNESS_ALIASES[n] ?? n;
+}
+/** Look up a registered adapter by `name` (alias-aware, e.g. `claude`). */
 function getAdapter(name) {
-    return exports.ADAPTERS.find((a) => a.name === name);
+    const canonical = normalizeHarnessName(name);
+    return exports.ADAPTERS.find((a) => a.name === canonical);
+}
+/**
+ * The adapter whose instruction file is `filename` (e.g. `AGENTS.md` → codex,
+ * `CLAUDE.md` → claude-code), if any. The per-spec disambiguation signal: a
+ * `<file>.spec.ts` compiles a `<file>` instruction file, so the filename names
+ * the harness more specifically than config/detect for THAT spec.
+ */
+function adapterForInstructionFile(filename) {
+    return exports.ADAPTERS.find((a) => a.layout.instructionFile === filename);
 }
 /**
  * Resolve the adapter for a command: an explicit `--harness <name>` wins (throws
@@ -42,7 +69,7 @@ function getAdapter(name) {
  * uses so detection + override live in one place.
  */
 function resolveAdapter(root, harness) {
-    if (harness !== undefined) {
+    if (harness !== undefined && harness !== "") {
         const a = getAdapter(harness);
         if (!a) {
             const known = exports.ADAPTERS.map((x) => x.name).join(", ");
@@ -52,4 +79,52 @@ function resolveAdapter(root, harness) {
     }
     return detectAdapter(root);
 }
+/** Normalize a config `harness` value (string | string[]) to a canonical list. */
+function normalizeHarnessList(harness) {
+    if (harness === undefined)
+        return [];
+    const arr = Array.isArray(harness) ? harness : [harness];
+    return arr.map(normalizeHarnessName).filter(Boolean);
+}
+/**
+ * Resolve the single harness a compile/lint operation should use, with explicit
+ * precedence — the deterministic replacement for sniffing the cwd:
+ *
+ *   1. `--harness=` flag (wins; throws if unknown).
+ *   2. config `harness` resolving to a single entry → use it.
+ *   3. config `harness` with multiple entries → use the first, with a loud notice.
+ *   4. no config → auto-detect, with a loud notice when the repo is ambiguous.
+ *
+ * `configHarness` is parsed once (alias-normalized) at the call site and passed
+ * in; this function re-normalizes idempotently so it's safe either way. Pure
+ * (besides reading `root`'s layout for detection) so the precedence is
+ * unit-testable without a real compile. See research/multi-harness-compile.md.
+ */
+function resolveHarnessSelection(opts) {
+    const { root, flag, configHarness } = opts;
+    if (flag !== undefined && flag !== "") {
+        return { kind: "ok", adapter: resolveAdapter(root, flag) };
+    }
+    const list = normalizeHarnessList(configHarness);
+    if (list.length === 1) {
+        return { kind: "ok", adapter: resolveAdapter(root, list[0]) };
+    }
+    if (list.length > 1) {
+        const adapter = resolveAdapter(root, list[0]);
+        return {
+            kind: "notice",
+            adapter,
+            notice: `repo targets ${list.join(", ")} — compiling for ${adapter.name}; override with --harness=`,
+        };
+    }
+    const det = detectAdapterResult(root);
+    if (det.ambiguousWith.length > 0) {
+        return {
+            kind: "notice",
+            adapter: det.adapter,
+            notice: `repo matches ${[det.adapter.name, ...det.ambiguousWith].join(", ")} — set "harness" in .vigilesrc.json or use --harness=`,
+        };
+    }
+    return { kind: "ok", adapter: det.adapter };
+}
 //# sourceMappingURL=adapter-registry.js.map

package/dist/adapters/claude-code/agent-runtime.d.ts

@@ -3,7 +3,7 @@
  *
  * A subagent declares an allowed-tools contract in its frontmatter (`tools:`).
  * But that field is documentation, not a hard runtime boundary (Claude Code
- * issue #54898): permissions are session-wide, a subagent inherits the parent
+ * #4740/#21460, SDK #172): permissions are session-wide, a subagent inherits the parent
  * session's grants, and `tools:` only filters what's *offered* — it can't deny
  * what the session allows. The deterministic layer that actually closes the gap
  * is a **PreToolUse hook** that blocks any tool the active agent's contract

package/dist/adapters/claude-code/agent-runtime.js

@@ -4,7 +4,7 @@
  *
  * A subagent declares an allowed-tools contract in its frontmatter (`tools:`).
  * But that field is documentation, not a hard runtime boundary (Claude Code
- * issue #54898): permissions are session-wide, a subagent inherits the parent
+ * #4740/#21460, SDK #172): permissions are session-wide, a subagent inherits the parent
  * session's grants, and `tools:` only filters what's *offered* — it can't deny
  * what the session allows. The deterministic layer that actually closes the gap
  * is a **PreToolUse hook** that blocks any tool the active agent's contract

package/dist/arg-match.d.ts

@@ -0,0 +1,28 @@
+/**
+ * `ArgMatcher` — a small, serializable matcher over a tool call's `input`.
+ *
+ * Shared by the check vocabulary (`toolWith` / `notTool` in `src/check.ts`) and
+ * the tool-interception seam (`src/tool-intercept.ts`), so "did the agent call
+ * this tool with these arguments?" means the same thing whether you're *asserting*
+ * on a captured call or *intercepting* one before it runs. Pure + model-free.
+ */
+/**
+ * A declarative matcher over a tool call's `input`, keyed by **dot-path** (e.g.
+ * `"body.prompt"`). Each value is matched against the value at that path: a
+ * `RegExp` is a pattern over the stringified value (use this for "contains"), and
+ * a `string`/`number`/`boolean` is an **exact** match (use this for "equals", e.g.
+ * a push target). All keys must match (AND). Serializable, so anything carrying
+ * one still round-trips through `toJSON`.
+ */
+export type ArgMatcher = Record<string, string | number | boolean | RegExp>;
+/** Render any tool-input value as a string for matching / messages. */
+export declare function stringifyValue(v: unknown): string;
+/** Resolve a dot-path (`"a.b.c"`) within an arbitrary value, or undefined. */
+export declare function getPath(obj: unknown, path: string): unknown;
+/** Does `input` satisfy every entry of `matcher`? (RegExp = pattern, else exact.) */
+export declare function matchesArgs(input: unknown, matcher: ArgMatcher): boolean;
+/** A human-readable form of a matcher for failure messages. */
+export declare function describeArgs(matcher: ArgMatcher): string;
+/** Serialize a matcher for `toJSON` (RegExp → its string form). */
+export declare function serializeArgs(matcher: ArgMatcher): Record<string, string | number | boolean>;
+//# sourceMappingURL=arg-match.d.ts.map

package/dist/arg-match.js

@@ -0,0 +1,61 @@
+"use strict";
+/**
+ * `ArgMatcher` — a small, serializable matcher over a tool call's `input`.
+ *
+ * Shared by the check vocabulary (`toolWith` / `notTool` in `src/check.ts`) and
+ * the tool-interception seam (`src/tool-intercept.ts`), so "did the agent call
+ * this tool with these arguments?" means the same thing whether you're *asserting*
+ * on a captured call or *intercepting* one before it runs. Pure + model-free.
+ */
+Object.defineProperty(exports, "__esModule", { value: true });
+exports.stringifyValue = stringifyValue;
+exports.getPath = getPath;
+exports.matchesArgs = matchesArgs;
+exports.describeArgs = describeArgs;
+exports.serializeArgs = serializeArgs;
+/** Render any tool-input value as a string for matching / messages. */
+function stringifyValue(v) {
+    if (typeof v === "string")
+        return v;
+    if (typeof v === "object" && v !== null) {
+        try {
+            return JSON.stringify(v) ?? "[object]";
+        }
+        catch {
+            return "[object]";
+        }
+    }
+    return String(v);
+}
+/** Resolve a dot-path (`"a.b.c"`) within an arbitrary value, or undefined. */
+function getPath(obj, path) {
+    let cur = obj;
+    for (const part of path.split(".")) {
+        if (cur === null || typeof cur !== "object")
+            return undefined;
+        cur = cur[part];
+    }
+    return cur;
+}
+/** Does `input` satisfy every entry of `matcher`? (RegExp = pattern, else exact.) */
+function matchesArgs(input, matcher) {
+    return Object.entries(matcher).every(([key, m]) => {
+        const value = getPath(input, key);
+        return m instanceof RegExp ? m.test(stringifyValue(value)) : value === m;
+    });
+}
+/** A human-readable form of a matcher for failure messages. */
+function describeArgs(matcher) {
+    return Object.entries(matcher)
+        .map(([k, m]) => `${k}=${m instanceof RegExp ? String(m) : JSON.stringify(m)}`)
+        .join(", ");
+}
+/** Serialize a matcher for `toJSON` (RegExp → its string form). */
+function serializeArgs(matcher) {
+    const out = {};
+    for (const [k, m] of Object.entries(matcher)) {
+        out[k] = m instanceof RegExp ? String(m) : m;
+    }
+    return out;
+}
+//# sourceMappingURL=arg-match.js.map

package/dist/check.d.ts

@@ -18,6 +18,8 @@
  */
 import type { Trace } from "./harness-test.js";
 import type { HookRunResult } from "./run-hook.js";
+import { type ArgMatcher } from "./arg-match.js";
+export type { ArgMatcher };
 /** The outcome of evaluating one check against one result. */
 export interface CheckResult {
     /** Did the check hold? */
@@ -54,6 +56,28 @@ export declare function evalChecks<T>(target: T, checks: readonly Check<T>[]): C
 export declare function assertChecks<T>(target: T, checks: readonly Check<T>[]): void;
 /** The agent invoked a tool by this name (regardless of result). */
 export declare function tool(name: string): Check<Trace>;
+/**
+ * The agent used tool `name` with at least one call whose `input` matches `args`
+ * — the **argument** half of the tool-call spy. Asserts not just *that* a tool was
+ * reached but *how* it was called (the image-request body carried the style suffix,
+ * the panel spawn requested a non-expert), which a completion grader can't see.
+ *
+ * Cross-reference: ≈ promptfoo `is-valid-function-call` / DeepEval `ToolCorrectnessMetric`.
+ */
+export declare function toolWith(name: string, args: ArgMatcher): Check<Trace>;
+/**
+ * The agent did NOT use tool `name` — the **safety / negative** assertion. With
+ * `args`, only calls whose `input` matches are forbidden (so "did not push to
+ * `main`" still allows pushing elsewhere; "no paid API call" forbids it outright).
+ * This is the highest-value, most-overlooked check, and the one a
+ * completion-grading eval structurally cannot make: it sees the agent's *decision*
+ * to act, not just its final text.
+ *
+ * Cross-reference: this negative/safety assertion of a *decision not to act* has
+ * no promptfoo / DeepEval / Inspect equivalent — they grade what the agent DID,
+ * not what it correctly refrained from doing.
+ */
+export declare function notTool(name: string, args?: ArgMatcher): Check<Trace>;
 /** A skill resolved to this id (`<plugin>:<skill>`) without erroring. */
 export declare function skill(id: string): Check<Trace>;
 /** The agent's final output contains a substring / matches a RegExp. */
@@ -112,8 +136,13 @@ interface UsageTrace {
     readonly usage: {
         readonly costUsd: number;
         readonly durationMs: number;
+        /** Fresh (uncached) input tokens, billed at full input price. */
         readonly inputTokens: number;
         readonly outputTokens: number;
+        /** Tokens written to the prompt cache this run (~1.25× input price). */
+        readonly cacheCreationTokens: number;
+        /** Tokens served from the prompt cache this run (~0.1× input price). */
+        readonly cacheReadTokens: number;
     };
 }
 /** The run cost at most `maxUsd`. */
@@ -128,5 +157,27 @@ export declare function latency(opts: {
 export declare function tokens(opts: {
     max: number;
 }): Check<UsageTrace>;
-export {};
+/**
+ * The run used at most `max` **fresh (uncached) input** tokens. The honest input
+ * side of a cost claim: a skill or CLAUDE.md injection adds input every turn, so
+ * a "compression" win on output can be erased here. (Cache reads are separate —
+ * see `cacheTokens`.)
+ */
+export declare function inputTokens(opts: {
+    max: number;
+}): Check<UsageTrace>;
+/** The run used at most `max` **output** tokens — the generation side. */
+export declare function outputTokens(opts: {
+    max: number;
+}): Check<UsageTrace>;
+/**
+ * Bound the prompt-cache token classes a run uses. `maxCreation` caps tokens
+ * **written** to the cache (the ~1.25× write premium — a fresh/cold prompt);
+ * `maxRead` caps tokens **served** from cache (~0.1× input). Each constraint is
+ * checked only when provided; the check passes when every provided bound holds.
+ */
+export declare function cacheTokens(opts: {
+    maxCreation?: number;
+    maxRead?: number;
+}): Check<UsageTrace>;
 //# sourceMappingURL=check.d.ts.map

package/dist/check.js

@@ -3,6 +3,8 @@ Object.defineProperty(exports, "__esModule", { value: true });
 exports.evalChecks = evalChecks;
 exports.assertChecks = assertChecks;
 exports.tool = tool;
+exports.toolWith = toolWith;
+exports.notTool = notTool;
 exports.skill = skill;
 exports.output = output;
 exports.hookFired = hookFired;
@@ -17,7 +19,11 @@ exports.judged = judged;
 exports.cost = cost;
 exports.latency = latency;
 exports.tokens = tokens;
+exports.inputTokens = inputTokens;
+exports.outputTokens = outputTokens;
+exports.cacheTokens = cacheTokens;
 const judge_js_1 = require("./judge.js");
+const arg_match_js_1 = require("./arg-match.js");
 /** Evaluate every check against a target. Pure — the shared core of `expect`
  * (strict) and `measure` (scored). */
 function evalChecks(target, checks) {
@@ -71,6 +77,63 @@ function tool(name) {
         toJSON: () => ({ kind: "tool", name }),
     };
 }
+/**
+ * The agent used tool `name` with at least one call whose `input` matches `args`
+ * — the **argument** half of the tool-call spy. Asserts not just *that* a tool was
+ * reached but *how* it was called (the image-request body carried the style suffix,
+ * the panel spawn requested a non-expert), which a completion grader can't see.
+ *
+ * Cross-reference: ≈ promptfoo `is-valid-function-call` / DeepEval `ToolCorrectnessMetric`.
+ */
+function toolWith(name, args) {
+    return {
+        kind: "toolWith",
+        eval: (t) => {
+            const calls = t.toolCalls.filter((c) => c.name === name);
+            const want = (0, arg_match_js_1.describeArgs)(args);
+            if (calls.length === 0) {
+                return no(`expected the agent to use tool "${name}" (with ${want}), but it used ${distinctToolNames(t.toolCalls)}`);
+            }
+            return calls.some((c) => (0, arg_match_js_1.matchesArgs)(c.input, args))
+                ? ok(`agent used "${name}" with ${want}`)
+                : no(`agent used "${name}" but never with ${want} (saw ${calls
+                    .map((c) => truncate((0, arg_match_js_1.stringifyValue)(c.input), 60))
+                    .join("; ")})`);
+        },
+        toJSON: () => ({ kind: "toolWith", name, args: (0, arg_match_js_1.serializeArgs)(args) }),
+    };
+}
+/**
+ * The agent did NOT use tool `name` — the **safety / negative** assertion. With
+ * `args`, only calls whose `input` matches are forbidden (so "did not push to
+ * `main`" still allows pushing elsewhere; "no paid API call" forbids it outright).
+ * This is the highest-value, most-overlooked check, and the one a
+ * completion-grading eval structurally cannot make: it sees the agent's *decision*
+ * to act, not just its final text.
+ *
+ * Cross-reference: this negative/safety assertion of a *decision not to act* has
+ * no promptfoo / DeepEval / Inspect equivalent — they grade what the agent DID,
+ * not what it correctly refrained from doing.
+ */
+function notTool(name, args) {
+    return {
+        kind: "notTool",
+        eval: (t) => {
+            const calls = t.toolCalls.filter((c) => c.name === name);
+            const offending = args
+                ? calls.filter((c) => (0, arg_match_js_1.matchesArgs)(c.input, args))
+                : calls;
+            const what = args ? `"${name}" with ${(0, arg_match_js_1.describeArgs)(args)}` : `"${name}"`;
+            const first = offending[0];
+            if (!first)
+                return ok(`agent never used ${what}`);
+            return no(`expected the agent NOT to use ${what}, but it did (${truncate((0, arg_match_js_1.stringifyValue)(first.input), 60)})`);
+        },
+        toJSON: () => args
+            ? { kind: "notTool", name, args: (0, arg_match_js_1.serializeArgs)(args) }
+            : { kind: "notTool", name },
+    };
+}
 /** A skill resolved to this id (`<plugin>:<skill>`) without erroring. */
 function skill(id) {
     return {
@@ -315,4 +378,62 @@ function tokens(opts) {
         toJSON: () => ({ kind: "tokens", max: opts.max }),
     };
 }
+/**
+ * The run used at most `max` **fresh (uncached) input** tokens. The honest input
+ * side of a cost claim: a skill or CLAUDE.md injection adds input every turn, so
+ * a "compression" win on output can be erased here. (Cache reads are separate —
+ * see `cacheTokens`.)
+ */
+function inputTokens(opts) {
+    return {
+        kind: "inputTokens",
+        eval: (t) => {
+            const v = t.usage.inputTokens;
+            return v <= opts.max
+                ? ok(`${String(v)} input tokens ≤ ${String(opts.max)}`)
+                : no(`expected ≤ ${String(opts.max)} input tokens, got ${String(v)}`);
+        },
+        toJSON: () => ({ kind: "inputTokens", max: opts.max }),
+    };
+}
+/** The run used at most `max` **output** tokens — the generation side. */
+function outputTokens(opts) {
+    return {
+        kind: "outputTokens",
+        eval: (t) => {
+            const v = t.usage.outputTokens;
+            return v <= opts.max
+                ? ok(`${String(v)} output tokens ≤ ${String(opts.max)}`)
+                : no(`expected ≤ ${String(opts.max)} output tokens, got ${String(v)}`);
+        },
+        toJSON: () => ({ kind: "outputTokens", max: opts.max }),
+    };
+}
+/**
+ * Bound the prompt-cache token classes a run uses. `maxCreation` caps tokens
+ * **written** to the cache (the ~1.25× write premium — a fresh/cold prompt);
+ * `maxRead` caps tokens **served** from cache (~0.1× input). Each constraint is
+ * checked only when provided; the check passes when every provided bound holds.
+ */
+function cacheTokens(opts) {
+    return {
+        kind: "cacheTokens",
+        eval: (t) => {
+            const created = t.usage.cacheCreationTokens;
+            const read = t.usage.cacheReadTokens;
+            if (opts.maxCreation !== undefined && created > opts.maxCreation) {
+                return no(`expected ≤ ${String(opts.maxCreation)} cache-creation tokens, got ${String(created)}`);
+            }
+            if (opts.maxRead !== undefined && read > opts.maxRead) {
+                return no(`expected ≤ ${String(opts.maxRead)} cache-read tokens, got ${String(read)}`);
+            }
+            return ok(`cache tokens within bounds (created ${String(created)}, read ${String(read)})`);
+        },
+        toJSON: () => ({
+            kind: "cacheTokens",
+            ...(opts.maxCreation !== undefined && { maxCreation: opts.maxCreation }),
+            ...(opts.maxRead !== undefined && { maxRead: opts.maxRead }),
+        }),
+    };
+}
 //# sourceMappingURL=check.js.map

package/dist/claude-code.d.ts

@@ -7,6 +7,7 @@
  */
 export * from "./adapters/claude-code/plugin-loader.js";
 export * from "./mock-model.js";
+export { claudeCodeDriver, buildClaudeArgs, parseClaudeRun, claudeAvailable, } from "./harness-test.js";
 export * from "./adapters/claude-code/dialect.js";
 export * from "./adapters/claude-code/layout.js";
 export * from "./adapters/claude-code/runtime.js";

package/dist/claude-code.js

@@ -14,6 +14,7 @@ var __exportStar = (this && this.__exportStar) || function(m, exports) {
     for (var p in m) if (p !== "default" && !Object.prototype.hasOwnProperty.call(exports, p)) __createBinding(exports, m, p);
 };
 Object.defineProperty(exports, "__esModule", { value: true });
+exports.claudeAvailable = exports.parseClaudeRun = exports.buildClaudeArgs = exports.claudeCodeDriver = void 0;
 /**
  * `vigiles/claude-code` — the Claude Code-specific harness pieces a *different*
  * harness would swap out: the plugin/repo loader (reads real Claude Code plugin
@@ -23,6 +24,16 @@ Object.defineProperty(exports, "__esModule", { value: true });
  */
 __exportStar(require("./adapters/claude-code/plugin-loader.js"), exports);
 __exportStar(require("./mock-model.js"), exports);
+// The Claude-Code harness-test transport — the default driver + its argv/parse
+// helpers + the `claude` capability probe. Agnostic users never need these
+// (`runHarnessTest` defaults to the CC driver), but they're exposed here — beside
+// the Codex driver in `vigiles/codex` — for CC-specific tests/tooling. They are
+// deliberately NOT on the agnostic `vigiles/testing` surface.
+var harness_test_js_1 = require("./harness-test.js");
+Object.defineProperty(exports, "claudeCodeDriver", { enumerable: true, get: function () { return harness_test_js_1.claudeCodeDriver; } });
+Object.defineProperty(exports, "buildClaudeArgs", { enumerable: true, get: function () { return harness_test_js_1.buildClaudeArgs; } });
+Object.defineProperty(exports, "parseClaudeRun", { enumerable: true, get: function () { return harness_test_js_1.parseClaudeRun; } });
+Object.defineProperty(exports, "claudeAvailable", { enumerable: true, get: function () { return harness_test_js_1.claudeAvailable; } });
 __exportStar(require("./adapters/claude-code/dialect.js"), exports);
 __exportStar(require("./adapters/claude-code/layout.js"), exports);
 __exportStar(require("./adapters/claude-code/runtime.js"), exports);