vigiles 5.0.0 → 5.1.0

package/README.md

@@ -5,12 +5,11 @@
 <h1 align="center">vigiles</h1>
 
 <p align="center">
-  <em>Quis custodiet ipsos custodes?</em> — Who watches the watchmen?
+  <strong>Lint &amp; test the harness your AI agent runs on.</strong>
 </p>
 
 <p align="center">
-  <strong>The missing linting + testing layer for agentic coding.</strong><br />
-  vigiles <strong>lints</strong> the references your instruction files make — linter rules, file paths, scripts, code symbols — and <strong>tests</strong> whether your hooks, skills, and CLAUDE.md actually change what the agent does.
+  Your CLAUDE.md, hooks, and skills steer the agent — but nothing checks they're <em>true</em>, and nothing tests they <em>work</em>. vigiles does both.
 </p>
 
 <p align="center">
@@ -22,130 +21,33 @@
 ---
 
 `Agent = Model + Harness`. You'd never ship an app without a linter and a test
-suite — yet an AI agent steering your repo is trusted on vibes. vigiles is the
-deterministic layer for the harness: it **lints** the references your instruction
-files make and **tests** that your hooks and skills actually fire. Two independent
-pillars — adopt either, or both:
+suite — yet the harness steering your agent runs on vibes. vigiles[^name] is the
+deterministic layer for it, and does two independent things — adopt either, or both:
 
-|       | Pillar                          | What it does                                                                                                                                                                                                                                                              |
-| ----- | ------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
-| **①** | **Lint your instruction files** | Every linter rule, file path, script, and code symbol your CLAUDE.md cites is checked against reality, so stale references can't silently mislead the agent. → [guide](docs/verifying-instruction-files.md)                                                               |
-| **②** | **Test your harness**           | Your hooks and skills are code — vigiles tests they actually fire, **deterministically and free** (no model, no API key); and when a question _does_ need a real-model eval, it runs on your **Claude subscription**, not metered API. → [guide](docs/harness-testing.md) |
+|             |                                                                                                                                                                                                                                                  |
+| ----------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
+| **🔎 Lint** | Every file path, script, code symbol, and linter rule your CLAUDE.md cites is checked against reality — so a renamed file or a disabled rule can't silently mislead the agent. **[→](docs/verifying-instruction-files.md)**                      |
+| **🧪 Test** | Hooks, skills, and subagents are code. vigiles tests they _do their job_ — and almost all of it is **deterministic, no API key**; the real-model evals run on your **Claude subscription**, not metered tokens. **[→](docs/harness-testing.md)** |
 
-Neither pillar depends on the other — pick the one that hurts today. **Works with
-Claude Code and Codex** ([`vigiles/codex`](docs/harnesses.md)) behind a five-port
-adapter; [custom adapters welcome](docs/authoring-an-adapter.md).
-
-## ① Lint — your CLAUDE.md lies to your agent
-
-Your CLAUDE.md says _"enforce `eslint/no-console`."_ But it was switched off
-months ago — and the agent trusts the claim. (Same story for the file path it
-cites that got renamed, and the script that was deleted.)
-
-**Without vigiles:** nobody checks. The agent acts on fiction.
-
-**With vigiles:** `npx vigiles lint` resolves every reference against reality —
-
-```text
-CLAUDE.md (inline mode):
-  ✗ line 1: Rule "eslint/no-console" exists but is disabled in eslint config
-  ✓ line 2: eslint/eqeqeq
-  ✗ line 3: Rule "no-consoel" not found in eslint. Did you mean: "eslint/no-console"?
-```
-
-It resolves rule names across **7 linter catalogs** — the rule exists **and is
-enabled** — and checks file paths, scripts, and code symbols the same way. Start
-with one comment, no new files:
-
-```md
-<!-- vigiles:enforce eslint/no-console "Route output through logger.ts" -->
-```
-
-Step up to a typed `.spec.ts` (compiled to CLAUDE.md, compiler-grade) when you
-want it. **[Full guide →](docs/verifying-instruction-files.md)**
-
-## ② Test — does your harness actually fire?
-
-A hook can be wired wrong, a skill's description can fail to trigger, injected
-context can never reach the model — silently, all passing a naive "did it run?"
-check.
-
-**Without vigiles:** you assume your `--no-verify` guard blocks. You don't know.
-
-**With vigiles:** a deterministic test proves it — no model, no API key,
-milliseconds:
-
-```typescript
-import { runHook } from "vigiles/testing";
-
-const r = runHook(guard, {
-  hook_event_name: "PreToolUse",
-  tool_name: "Bash",
-  tool_input: { command: "git commit --no-verify" },
-});
-assert(r.blocked); // a red ✗ here means your hook silently lets it through
-```
-
-```text
-  ✓ guard blocks --no-verify and allows a clean commit
-
-2 passed.
-```
-
-Three tiers, cheapest first: **`runHook`** (a hook's logic), **`runHarnessTest`**
-(the real agent CLI against a scripted mock model), and the real-model scored tier
-(**`measure`** / **`runEval`**). **Testing a skill?** Two questions, both covered:
-does its description **fire** (`measureTriggerRate` — recall across varied prompts
-without hijacking unrelated ones, precision), **and** does its guidance actually
-**work**. For "is this exact skill any good?" score the output directly —
-`measure({ checks: [judged(rubric)] })` + `assertRates` (the **absolute** oracle,
-what promptfoo/DeepEval lead with; no on/off baseline needed). When you need the
-**relative** lift over no-skill — regression, or proving the change isn't noise —
-A/B it on-vs-off with `runEval` + `assertSignificant`. Description _and_ behavior,
-not just one. **Need a safety property** — that the agent
-**didn't** push to the wrong branch or call a paid API? `notTool` + `interceptTools`
-intercept the tool in the real hook layer, so the attempt is caught and the side
-effect never happens. **[Full guide →](docs/harness-testing.md)** · vigiles runs
-foreign code (and a real model) safely by default — **[safety model →](docs/safety.md)**
-
-Most of what real plugins do is testable cheaply — fire / trigger / contract /
-safety, plus **record-replay** for the tool/API results a skill consumes (recorded
-once from the real tool, replayed deterministically — no live service, no Docker).
-That covers ~90%+ of real plugin surface on your subscription; the rare case that
-needs a real browser or database **composes with Docker** rather than us
-reinventing the sandbox. **[What we test, how →](research/eval-coverage-and-isolation.md)**
-
-**Affordable by design — the eval you can actually run.** Almost nobody evals
-their harness, because the usual tools (promptfoo, DeepEval, …) hit the API SDK
-and bill **per token on every run**. vigiles inverts that: most questions are
-answered with **no model at all** (free, every commit), and when you do reach for
-a real-model eval, vigiles drives your `claude` CLI — so it runs on the **Pro/Max
-subscription you already pay for**, not metered API billing. CI runs only the free
-deterministic tiers; you run the real-model eval where the subscription already is
-— a Claude Code session or locally — when it's worth it, not on every PR.
-
-This affordability story is **ToS-clean**: vigiles drives _your own_ `claude` CLI
-to test _your own_ harness on _your own_ subscription — the same thing you do when
-you run Claude Code. (The Claude Agent SDK's ToS restricts _productizing_ claude.ai
-login/limits in a third-party offering; running your own tests on your own sub is
-exactly the supported posture, not that.)
+Pick the one that hurts today. **Works with Claude Code and Codex**
+([`vigiles/codex`](docs/harnesses.md)), and you can
+[teach it your own harness](docs/authoring-an-adapter.md).
 
 ## Quick start
 
 **Paste into Claude Code or Codex:**
 
 ```text
-Install vigiles in this repo and run it. Verify my CLAUDE.md / AGENTS.md
-references and show me what's stale, then write and run a harness test for one
-of my hooks or skills. Use good defaults (both pillars, non-interactive), but
-ask me first whether to gate it in CI, whether to add a real-model eval, and
-whether to enforce strictly (--strict).
+Set up vigiles in this repo with good defaults (lint + test, non-interactive).
+Verify my CLAUDE.md / AGENTS.md references and show me what's stale, then write
+and run a harness test for one of my hooks or skills. Ask me first before gating
+it in CI, adding a real-model eval, or enforcing strictly (--strict).
 ```
 
 Or do it yourself:
 
 ```bash
-npx vigiles init   # sets up BOTH pillars: spec + harness test + CI + plugin
+npx vigiles init   # sets up lint + test: spec + harness test + CI + plugin
 ```
 
 It's interactive in a terminal and non-interactive for agents/CI (or with
@@ -156,7 +58,7 @@ tell your agent _"test my skills"_ and it picks the tier and writes the test.
 <details>
 <summary>What <code>init</code> sets up</summary>
 
-- **Both pillars** by default; scope with `--lint` / `--test` (one or both).
+- **Both lint and test** by default; scope with `--lint` / `--test` (one or both).
 - Adds `vigiles` to your `devDependencies`.
 - Installs the Claude Code plugin (skills + hooks) via the marketplace —
   globally, never vendored into your repo.
@@ -172,9 +74,71 @@ Prefer to write tests yourself? They can be JS **or** TS
 
 </details>
 
+## ① Lint — your CLAUDE.md lies to your agent
+
+Your CLAUDE.md points the agent at `src/auth/login.ts` and tells it to run
+`npm run check`. But the file moved to `src/auth/session.ts` six commits ago, and
+the script was renamed. The agent trusts the stale claim and acts on fiction.
+
+`npx vigiles lint` resolves every reference against reality:
+
+```text
+CLAUDE.md:
+  ✗ src/auth/login.ts — no such file (renamed or moved?)
+  ✗ npm run check — not in package.json. Did you mean: "check:types"?
+  ✓ @typescript-eslint/no-floating-promises — exists and enabled in eslint config
+```
+
+File paths, scripts, and code symbols — plus linter rules across **7 catalogs**
+(the rule exists **and is enabled**). Start with one inline comment, no new files;
+step up to a typed `.spec.ts` (compiled to CLAUDE.md, compiler-grade) when you want
+it. **[Full guide →](docs/verifying-instruction-files.md)**
+
+**Same cross-reference, any plugin.** `npx vigiles scan` checks a plugin's
+_contracts_ — every subagent tool, `mcp__server__tool`, `mcp_tool` hook, hook
+event, and script path actually **exists and resolves**, not just parses (valid
+YAML ≠ a tool that's real). A **superset of Anthropic's `claude plugin validate`**,
+no key. **[Audit any plugin →](docs/cli.md#scan-dir)**
+
+## ② Test — does your harness do its job?
+
+A hook can be wired wrong. A skill's description can fail to trigger — or hijack
+unrelated prompts. Injected context can never reach the model. All of it passes a
+naive "did it run?" check. vigiles tests the assembled harness for real:
+
+```typescript
+import { runHook } from "vigiles/testing";
+
+const r = runHook(guard, {
+  hook_event_name: "PreToolUse",
+  tool_name: "Bash",
+  tool_input: { command: "git commit --no-verify" },
+});
+assert(r.blocked); // a red ✗ means your guard silently lets it through
+```
+
+It goes well past _"did it fire?"_:
+
+- **Hooks block** what they must — `runHook`, or the real agent CLI via `runHarnessTest`.
+- **Skills trigger** on the right prompts and stay quiet on the wrong ones — recall _and_ precision (`measureTriggerRate`).
+- **Behaviour is good** — score a skill's output directly, or A/B it on-vs-off for the real lift over no-skill (`measure` / `runEval`, with significance testing).
+- **Safety holds** — the agent _didn't_ push to the wrong branch or hit a paid API; `interceptTools` catches the attempt so the side effect never happens.
+
+**The eval you can actually afford.** Almost every tier runs with **no model and
+no API key** — milliseconds, on every commit. The rest drive your own `claude` CLI:
+
+|                        | Runs on                 | Cost                                        |
+| ---------------------- | ----------------------- | ------------------------------------------- |
+| promptfoo, DeepEval, … | metered API SDK         | billed **per token, every run**             |
+| **vigiles**            | your Claude Pro/Max sub | **$0 extra** — and most tiers need no model |
+
+That's why you can eval your harness on every change, not just once.
+**[How it works →](docs/harness-testing.md)** · **[Why it's affordable →](docs/eval-architecture.md)** · **[Safety model →](docs/safety.md)**
+
 ## More
 
-- **[CLI & GitHub Action →](docs/cli.md)** — every command, the Action (inputs / output / versioning), the Claude Code plugin, and the five `lint` rules.
+- **[Plugin health leaderboard →](docs/cli.md#scan-dir)** — point `scan` at a marketplace (e.g. `wshobson/agents`) and it ranks every plugin by structural health (0–100, A–F), worst issues first — still **no key**. Add `--trigger` for the model-gated column: do the skills actually fire?
+- **[CLI & GitHub Action →](docs/cli.md)** — every command, the Action (inputs / output / versioning), the Claude Code plugin, and the `lint` rules.
 - **[Skills →](docs/skills.md)** — consumer skills installed as a Claude Code plugin: `/plugin marketplace add zernie/vigiles` then `/plugin install vigiles@vigiles` (or let `vigiles init` do it). The model-invocable ones (`test-harness`, `strengthen`, `edit-spec`) fire on their own — ask _"test my skills"_, _"strengthen my rules"_, or _"add a rule to CLAUDE.md"_ and the agent reaches for them; `migrate-to-spec` and `linter-docs` are user-invoked.
 - **[Docs index →](docs/README.md)** · **[Research →](research/README.md)** · **[Related tools →](docs/related-tools.md)** (ast-grep, Dependency Cruiser, Ruler, rulesync).
 - Companion to [Feedback Loop Is All You Need](https://zernie.com/blog/feedback-loop-is-all-you-need).
@@ -182,3 +146,5 @@ Prefer to write tests yourself? They can be JS **or** TS
 ## License
 
 [MIT](LICENSE)
+
+[^name]: **vigiles** — the watchmen of ancient Rome, who guarded the city (and fought its fires) by night. _Quis custodiet ipsos custodes?_ — "who watches the watchmen?" (Juvenal, _Satire VI_).

package/dist/adapters/claude-code/agent-runtime.d.ts

@@ -31,6 +31,16 @@
  * an empty list (`tools:` with nothing after it) means "no tools allowed".
  */
 export declare function parseAgentTools(markdown: string): string[] | null;
+/**
+ * Parse a comma/array tool list under an arbitrary frontmatter `key` (e.g.
+ * `tools:` or `disallowedTools:`) via the shared lenient reader
+ * (core/frontmatter-read.ts): a real YAML parse (so `key: [Read, "Bash"]` is a
+ * native array and `key: Read, Bash` a comma scalar) with a regex salvage when
+ * the block is malformed — the rail still reads the contract. `null` when the key
+ * is absent (inherits all), `[]` when present-but-empty (no tools). Shared by the
+ * rail (`tools:`) and the `disallowed-tools-contract` scan/lint.
+ */
+export declare function parseAgentToolList(markdown: string, key: string): string[] | null;
 export interface PreToolDecision {
     /** Whether the tool call is allowed (true) or blocked (false). */
     readonly allow: boolean;

package/dist/adapters/claude-code/agent-runtime.js

@@ -25,6 +25,7 @@
  */
 Object.defineProperty(exports, "__esModule", { value: true });
 exports.parseAgentTools = parseAgentTools;
+exports.parseAgentToolList = parseAgentToolList;
 exports.decidePreToolUse = decidePreToolUse;
 exports.setActiveAgent = setActiveAgent;
 exports.clearActiveAgent = clearActiveAgent;
@@ -32,28 +33,10 @@ exports.readActiveAgent = readActiveAgent;
 exports.evaluatePreToolUse = evaluatePreToolUse;
 const node_fs_1 = require("node:fs");
 const node_path_1 = require("node:path");
+const frontmatter_read_js_1 = require("../../core/frontmatter-read.js");
 // ---------------------------------------------------------------------------
 // Parse the tool contract from a compiled agent .md
 // ---------------------------------------------------------------------------
-/** Extract the YAML frontmatter block (between the first pair of `---` fences). */
-function extractFrontmatter(markdown) {
-    const lines = markdown.split("\n");
-    let start = -1;
-    for (let i = 0; i < lines.length; i++) {
-        if (lines[i].trim() === "---") {
-            start = i;
-            break;
-        }
-    }
-    if (start === -1)
-        return null;
-    for (let i = start + 1; i < lines.length; i++) {
-        if (lines[i].trim() === "---") {
-            return lines.slice(start + 1, i).join("\n");
-        }
-    }
-    return null;
-}
 /**
  * Parse an agent's allowed-tools contract from its compiled markdown.
  *
@@ -63,16 +46,19 @@ function extractFrontmatter(markdown) {
  * an empty list (`tools:` with nothing after it) means "no tools allowed".
  */
 function parseAgentTools(markdown) {
-    const fm = extractFrontmatter(markdown);
-    if (fm === null)
-        return null;
-    const match = /^tools:[ \t]*(.*)$/m.exec(fm);
-    if (!match)
-        return null;
-    return match[1]
-        .split(",")
-        .map((t) => t.trim())
-        .filter((t) => t.length > 0);
+    return parseAgentToolList(markdown, "tools");
+}
+/**
+ * Parse a comma/array tool list under an arbitrary frontmatter `key` (e.g.
+ * `tools:` or `disallowedTools:`) via the shared lenient reader
+ * (core/frontmatter-read.ts): a real YAML parse (so `key: [Read, "Bash"]` is a
+ * native array and `key: Read, Bash` a comma scalar) with a regex salvage when
+ * the block is malformed — the rail still reads the contract. `null` when the key
+ * is absent (inherits all), `[]` when present-but-empty (no tools). Shared by the
+ * rail (`tools:`) and the `disallowed-tools-contract` scan/lint.
+ */
+function parseAgentToolList(markdown, key) {
+    return (0, frontmatter_read_js_1.frontmatterList)((0, frontmatter_read_js_1.readFrontmatter)(markdown), key);
 }
 /**
  * Decide whether `tool` is allowed under an agent's tool contract. Pure, so the

package/dist/adapters/claude-code/dialect.js

@@ -9,9 +9,13 @@ exports.claudeCodeDialect = {
         "Read",
         "Write",
         "Edit",
+        "MultiEdit",
         "Bash",
+        "BashOutput",
+        "KillBash",
         "Grep",
         "Glob",
+        "LS",
         "WebSearch",
         "WebFetch",
         "NotebookEdit",
@@ -30,12 +34,24 @@ exports.claudeCodeDialect = {
         "WaitForMcpServers",
     ],
     mcpToolPattern: /^mcp__[a-z0-9_-]+__[a-z0-9_-]+$/i,
+    // Claude Code's own built-in MCP server: the IDE integration provides
+    // `mcp__ide__getDiagnostics` / `mcp__ide__executeCode` at runtime without any
+    // plugin declaring it, so a contract that lists those must NOT be flagged as
+    // referencing an undeclared server (the mcp-tool-resolves allowlist).
+    knownMcpServers: ["ide"],
+    // The real Claude Code hook events. (Was wrong: PreSession/PostSession don't
+    // exist; SessionStart/SessionEnd/Stop/SubagentStop/UserPromptSubmit/PreCompact
+    // were missing — verified against the events real plugins register.)
     hookEvents: [
         "PreToolUse",
         "PostToolUse",
-        "PreSession",
-        "PostSession",
+        "UserPromptSubmit",
         "Notification",
+        "Stop",
+        "SubagentStop",
+        "PreCompact",
+        "SessionStart",
+        "SessionEnd",
     ],
     // Claude Code natively reads CLAUDE.md only — it does NOT auto-load AGENTS.md
     // (anthropics/claude-code#34235 is open; AGENTS.md works solely via an

package/dist/adapters/codex/eval.d.ts

@@ -0,0 +1,94 @@
+/**
+ * Codex EVAL-tier transport — the runner + trace parser that
+ * `measureTriggerRate`/`runEval` dispatch to via the `ModelOutputParser` seam.
+ *
+ * SCHEMA: CONFIRMED against real `codex exec --json` (codex-cli 0.139.0, ChatGPT
+ * auth). The stream is the thread/item model:
+ *
+ *   {"type":"thread.started","thread_id":"…"}
+ *   {"type":"turn.started"}
+ *   {"type":"item.started","item":{"id":"item_0","type":"command_execution",…}}   // mid-flight
+ *   {"type":"item.completed","item":{"id":"item_0","type":"command_execution","command":"…","aggregated_output":"…","exit_code":0}}
+ *   {"type":"item.completed","item":{"id":"item_1","type":"agent_message","text":"…"}}
+ *   {"type":"turn.completed","usage":{"input_tokens":…,"cached_input_tokens":…,"output_tokens":…}}
+ *
+ * So: assistant text = `item.completed` with `item.type:"agent_message"` →
+ * `item.text`; a tool call = `item.type:"command_execution"` → `item.command`;
+ * usage rides `turn.completed`. We count `item.completed` ONLY (an `item.started`
+ * carries the same `id` mid-flight — counting both double-counts).
+ *
+ * THE SKILL FINDING: Codex has NO discrete "skill selected" event (its CLI has no
+ * Skill-tool concept). When a skill triggers, the model READS the skill's
+ * `SKILL.md` via a `command_execution` (`sed/cat … skills/<name>/SKILL.md`) and
+ * usually says so in an `agent_message`. So "did skill X fire" on Codex is not a
+ * clean trace event like Claude's `Skill` tool_use — it's detected by the
+ * SKILL.md read (`codexSkillFired`). Best-effort by nature (a cached skill might
+ * not be re-read); pair with a behavioral/judged check for certainty.
+ */
+import type { ParsedModelRun, AgentRunArgs, RunOut, EvalDriver } from "../../eval.js";
+import type { ToolCall } from "../../core/harness-driver.js";
+/** Parse `codex exec --json` stdout into the common trace fields (confirmed schema). */
+export declare function parseCodexEvalRun(out: {
+    stdout: string;
+}): ParsedModelRun;
+/**
+ * The error message if the run errored or was rate-limited (an `error` /
+ * `turn.failed` event), else null. CRITICAL for the eval tier: an errored turn
+ * must NOT be scored as a clean "skill didn't fire" miss — dogfooding hit a Codex
+ * usage limit ("You've hit your usage limit…") whose `error` event left an empty
+ * trace that `codexSkillFired` read as recall 0. A caller should skip/retry an
+ * errored run, not count it. (The Claude path has `isRateLimited` + backoff; this
+ * is the Codex equivalent detector.)
+ */
+export declare function codexRunError(out: {
+    stdout: string;
+}): string | null;
+/**
+ * Did Codex activate skill `name` on this run? Detected by the SKILL.md read —
+ * Codex has no discrete skill-selection event, so when a skill triggers the model
+ * reads its `…/<name>/SKILL.md` via a `command_execution`. Best-effort (a cached
+ * skill might not be re-read); for the trigger-rate `fired` predicate over Codex.
+ */
+export declare function codexSkillFired(run: {
+    toolCalls: readonly ToolCall[];
+}, name: string): boolean;
+/**
+ * Materialize a (Claude-shaped) plugin dir's skills into `<cwd>/.codex/skills/` —
+ * where Codex actually discovers them (validated live: codex reads
+ * `<cwd>/.codex/skills/<name>/SKILL.md`). This is the Codex analog of Claude's
+ * `--plugin-dir`: `measureTriggerRate` hands the runner a `pluginDir` (the
+ * stubbed/packaged skills), and the Codex runner installs them here before the
+ * turn. Pure fs — unit-testable without a binary.
+ */
+export declare function installCodexSkills(pluginDir: string, cwd: string): number;
+/**
+ * The Codex eval-tier `AgentRunner`: install the run's skills into `.codex/skills`
+ * (Codex's discovery path, vs Claude's `--plugin-dir`), then drive a real
+ * `codex exec --json` turn. The seam `measureTriggerRate(spec, { evalDriver:
+ * codexEvalDriver })` dispatches through.
+ */
+export declare function codexEvalAgentRunner(args: AgentRunArgs): Promise<RunOut>;
+/**
+ * The Codex eval driver — pass to `measureTriggerRate(spec, { evalDriver:
+ * codexEvalDriver })` to run a trigger-rate eval natively on `codex exec`. Pair
+ * the spec's `fired` with `codexSkillFired` (Codex has no Skill-tool event).
+ */
+export declare const codexEvalDriver: EvalDriver;
+/**
+ * Spawn real `codex exec --json` for the eval tier (real model, the user's codex
+ * auth — NOT the mock). CONFIRMED flags (codex 0.139.0): `--json` for the event
+ * stream, `--skip-git-repo-check` for a bare cwd, the approvals/sandbox bypass so
+ * the turn runs unattended, `-C <cwd>` for the working dir, prompt as the trailing
+ * positional, and stdin = /dev/null (`stdio: ["ignore",…]`) — codex otherwise
+ * blocks on "Reading additional input from stdin…". Needs ChatGPT/API auth +
+ * network egress to the model backend.
+ */
+export declare function codexEvalRunner(args: {
+    task: string;
+    cwd: string;
+    timeoutMs: number;
+}): {
+    code: number;
+    stdout: string;
+};
+//# sourceMappingURL=eval.d.ts.map