vigiles 5.0.1 β†’ 5.1.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (38) hide show
  1. package/README.md +14 -8
  2. package/dist/adapters/claude-code/agent-runtime.d.ts +10 -0
  3. package/dist/adapters/claude-code/agent-runtime.js +15 -29
  4. package/dist/adapters/claude-code/dialect.js +18 -2
  5. package/dist/adapters/codex/eval.d.ts +94 -0
  6. package/dist/adapters/codex/eval.js +227 -0
  7. package/dist/cli.js +464 -8
  8. package/dist/codex.d.ts +1 -0
  9. package/dist/codex.js +3 -0
  10. package/dist/core/compile.js +8 -36
  11. package/dist/core/description-overlap.d.ts +27 -0
  12. package/dist/core/description-overlap.js +53 -0
  13. package/dist/core/dialect.d.ts +8 -0
  14. package/dist/core/frontmatter-read.d.ts +25 -0
  15. package/dist/core/frontmatter-read.js +138 -0
  16. package/dist/core/hook-events.d.ts +34 -0
  17. package/dist/core/hook-events.js +48 -0
  18. package/dist/core/mcp-config.d.ts +20 -0
  19. package/dist/core/mcp-config.js +40 -0
  20. package/dist/core/mcp-hook.d.ts +35 -0
  21. package/dist/core/mcp-hook.js +70 -0
  22. package/dist/core/mcp-tool.d.ts +50 -0
  23. package/dist/core/mcp-tool.js +61 -0
  24. package/dist/core/tool-contract.d.ts +68 -0
  25. package/dist/core/tool-contract.js +113 -0
  26. package/dist/core/types.d.ts +89 -0
  27. package/dist/core/validate.js +22 -0
  28. package/dist/eval.d.ts +69 -13
  29. package/dist/eval.js +106 -51
  30. package/dist/leaderboard.js +61 -3
  31. package/dist/plugin-loader.d.ts +1 -0
  32. package/dist/plugin-loader.js +71 -18
  33. package/dist/scan-behavioral.d.ts +73 -0
  34. package/dist/scan-behavioral.js +150 -0
  35. package/dist/scan.d.ts +126 -1
  36. package/dist/scan.js +559 -40
  37. package/package.json +1 -1
  38. package/skills/migrate-to-spec/SKILL.md +0 -2
package/README.md CHANGED
@@ -30,19 +30,18 @@ deterministic layer for it, and does two independent things β€” adopt either, or
30
30
  | **πŸ§ͺ Test** | Hooks, skills, and subagents are code. vigiles tests they _do their job_ β€” and almost all of it is **deterministic, no API key**; the real-model evals run on your **Claude subscription**, not metered tokens. **[β†’](docs/harness-testing.md)** |
31
31
 
32
32
  Pick the one that hurts today. **Works with Claude Code and Codex**
33
- ([`vigiles/codex`](docs/harnesses.md)) behind a five-port adapter;
34
- [custom adapters welcome](docs/authoring-an-adapter.md).
33
+ ([`vigiles/codex`](docs/harnesses.md)), and you can
34
+ [teach it your own harness](docs/authoring-an-adapter.md).
35
35
 
36
36
  ## Quick start
37
37
 
38
38
  **Paste into Claude Code or Codex:**
39
39
 
40
40
  ```text
41
- Install vigiles in this repo and run it. Verify my CLAUDE.md / AGENTS.md
42
- references and show me what's stale, then write and run a harness test for one
43
- of my hooks or skills. Use good defaults (lint + test, non-interactive), but
44
- ask me first whether to gate it in CI, whether to add a real-model eval, and
45
- whether to enforce strictly (--strict).
41
+ Set up vigiles in this repo with good defaults (lint + test, non-interactive).
42
+ Verify my CLAUDE.md / AGENTS.md references and show me what's stale, then write
43
+ and run a harness test for one of my hooks or skills. Ask me first before gating
44
+ it in CI, adding a real-model eval, or enforcing strictly (--strict).
46
45
  ```
47
46
 
48
47
  Or do it yourself:
@@ -95,6 +94,12 @@ File paths, scripts, and code symbols β€” plus linter rules across **7 catalogs*
95
94
  step up to a typed `.spec.ts` (compiled to CLAUDE.md, compiler-grade) when you want
96
95
  it. **[Full guide β†’](docs/verifying-instruction-files.md)**
97
96
 
97
+ **Same cross-reference, any plugin.** `npx vigiles scan` checks a plugin's
98
+ _contracts_ β€” every subagent tool, `mcp__server__tool`, `mcp_tool` hook, hook
99
+ event, and script path actually **exists and resolves**, not just parses (valid
100
+ YAML β‰  a tool that's real). A **superset of Anthropic's `claude plugin validate`**,
101
+ no key. **[Audit any plugin β†’](docs/cli.md#scan-dir)**
102
+
98
103
  ## β‘‘ Test β€” does your harness do its job?
99
104
 
100
105
  A hook can be wired wrong. A skill's description can fail to trigger β€” or hijack
@@ -132,7 +137,8 @@ That's why you can eval your harness on every change, not just once.
132
137
 
133
138
  ## More
134
139
 
135
- - **[CLI & GitHub Action β†’](docs/cli.md)** β€” every command, the Action (inputs / output / versioning), the Claude Code plugin, and the five `lint` rules.
140
+ - **[Plugin health leaderboard β†’](docs/cli.md#scan-dir)** β€” point `scan` at a marketplace (e.g. `wshobson/agents`) and it ranks every plugin by structural health (0–100, A–F), worst issues first β€” still **no key**. Add `--trigger` for the model-gated column: do the skills actually fire?
141
+ - **[CLI & GitHub Action β†’](docs/cli.md)** β€” every command, the Action (inputs / output / versioning), the Claude Code plugin, and the `lint` rules.
136
142
  - **[Skills β†’](docs/skills.md)** β€” consumer skills installed as a Claude Code plugin: `/plugin marketplace add zernie/vigiles` then `/plugin install vigiles@vigiles` (or let `vigiles init` do it). The model-invocable ones (`test-harness`, `strengthen`, `edit-spec`) fire on their own β€” ask _"test my skills"_, _"strengthen my rules"_, or _"add a rule to CLAUDE.md"_ and the agent reaches for them; `migrate-to-spec` and `linter-docs` are user-invoked.
137
143
  - **[Docs index β†’](docs/README.md)** Β· **[Research β†’](research/README.md)** Β· **[Related tools β†’](docs/related-tools.md)** (ast-grep, Dependency Cruiser, Ruler, rulesync).
138
144
  - Companion to [Feedback Loop Is All You Need](https://zernie.com/blog/feedback-loop-is-all-you-need).
@@ -31,6 +31,16 @@
31
31
  * an empty list (`tools:` with nothing after it) means "no tools allowed".
32
32
  */
33
33
  export declare function parseAgentTools(markdown: string): string[] | null;
34
+ /**
35
+ * Parse a comma/array tool list under an arbitrary frontmatter `key` (e.g.
36
+ * `tools:` or `disallowedTools:`) via the shared lenient reader
37
+ * (core/frontmatter-read.ts): a real YAML parse (so `key: [Read, "Bash"]` is a
38
+ * native array and `key: Read, Bash` a comma scalar) with a regex salvage when
39
+ * the block is malformed β€” the rail still reads the contract. `null` when the key
40
+ * is absent (inherits all), `[]` when present-but-empty (no tools). Shared by the
41
+ * rail (`tools:`) and the `disallowed-tools-contract` scan/lint.
42
+ */
43
+ export declare function parseAgentToolList(markdown: string, key: string): string[] | null;
34
44
  export interface PreToolDecision {
35
45
  /** Whether the tool call is allowed (true) or blocked (false). */
36
46
  readonly allow: boolean;
@@ -25,6 +25,7 @@
25
25
  */
26
26
  Object.defineProperty(exports, "__esModule", { value: true });
27
27
  exports.parseAgentTools = parseAgentTools;
28
+ exports.parseAgentToolList = parseAgentToolList;
28
29
  exports.decidePreToolUse = decidePreToolUse;
29
30
  exports.setActiveAgent = setActiveAgent;
30
31
  exports.clearActiveAgent = clearActiveAgent;
@@ -32,28 +33,10 @@ exports.readActiveAgent = readActiveAgent;
32
33
  exports.evaluatePreToolUse = evaluatePreToolUse;
33
34
  const node_fs_1 = require("node:fs");
34
35
  const node_path_1 = require("node:path");
36
+ const frontmatter_read_js_1 = require("../../core/frontmatter-read.js");
35
37
  // ---------------------------------------------------------------------------
36
38
  // Parse the tool contract from a compiled agent .md
37
39
  // ---------------------------------------------------------------------------
38
- /** Extract the YAML frontmatter block (between the first pair of `---` fences). */
39
- function extractFrontmatter(markdown) {
40
- const lines = markdown.split("\n");
41
- let start = -1;
42
- for (let i = 0; i < lines.length; i++) {
43
- if (lines[i].trim() === "---") {
44
- start = i;
45
- break;
46
- }
47
- }
48
- if (start === -1)
49
- return null;
50
- for (let i = start + 1; i < lines.length; i++) {
51
- if (lines[i].trim() === "---") {
52
- return lines.slice(start + 1, i).join("\n");
53
- }
54
- }
55
- return null;
56
- }
57
40
  /**
58
41
  * Parse an agent's allowed-tools contract from its compiled markdown.
59
42
  *
@@ -63,16 +46,19 @@ function extractFrontmatter(markdown) {
63
46
  * an empty list (`tools:` with nothing after it) means "no tools allowed".
64
47
  */
65
48
  function parseAgentTools(markdown) {
66
- const fm = extractFrontmatter(markdown);
67
- if (fm === null)
68
- return null;
69
- const match = /^tools:[ \t]*(.*)$/m.exec(fm);
70
- if (!match)
71
- return null;
72
- return match[1]
73
- .split(",")
74
- .map((t) => t.trim())
75
- .filter((t) => t.length > 0);
49
+ return parseAgentToolList(markdown, "tools");
50
+ }
51
+ /**
52
+ * Parse a comma/array tool list under an arbitrary frontmatter `key` (e.g.
53
+ * `tools:` or `disallowedTools:`) via the shared lenient reader
54
+ * (core/frontmatter-read.ts): a real YAML parse (so `key: [Read, "Bash"]` is a
55
+ * native array and `key: Read, Bash` a comma scalar) with a regex salvage when
56
+ * the block is malformed β€” the rail still reads the contract. `null` when the key
57
+ * is absent (inherits all), `[]` when present-but-empty (no tools). Shared by the
58
+ * rail (`tools:`) and the `disallowed-tools-contract` scan/lint.
59
+ */
60
+ function parseAgentToolList(markdown, key) {
61
+ return (0, frontmatter_read_js_1.frontmatterList)((0, frontmatter_read_js_1.readFrontmatter)(markdown), key);
76
62
  }
77
63
  /**
78
64
  * Decide whether `tool` is allowed under an agent's tool contract. Pure, so the
@@ -9,9 +9,13 @@ exports.claudeCodeDialect = {
9
9
  "Read",
10
10
  "Write",
11
11
  "Edit",
12
+ "MultiEdit",
12
13
  "Bash",
14
+ "BashOutput",
15
+ "KillBash",
13
16
  "Grep",
14
17
  "Glob",
18
+ "LS",
15
19
  "WebSearch",
16
20
  "WebFetch",
17
21
  "NotebookEdit",
@@ -30,12 +34,24 @@ exports.claudeCodeDialect = {
30
34
  "WaitForMcpServers",
31
35
  ],
32
36
  mcpToolPattern: /^mcp__[a-z0-9_-]+__[a-z0-9_-]+$/i,
37
+ // Claude Code's own built-in MCP server: the IDE integration provides
38
+ // `mcp__ide__getDiagnostics` / `mcp__ide__executeCode` at runtime without any
39
+ // plugin declaring it, so a contract that lists those must NOT be flagged as
40
+ // referencing an undeclared server (the mcp-tool-resolves allowlist).
41
+ knownMcpServers: ["ide"],
42
+ // The real Claude Code hook events. (Was wrong: PreSession/PostSession don't
43
+ // exist; SessionStart/SessionEnd/Stop/SubagentStop/UserPromptSubmit/PreCompact
44
+ // were missing β€” verified against the events real plugins register.)
33
45
  hookEvents: [
34
46
  "PreToolUse",
35
47
  "PostToolUse",
36
- "PreSession",
37
- "PostSession",
48
+ "UserPromptSubmit",
38
49
  "Notification",
50
+ "Stop",
51
+ "SubagentStop",
52
+ "PreCompact",
53
+ "SessionStart",
54
+ "SessionEnd",
39
55
  ],
40
56
  // Claude Code natively reads CLAUDE.md only β€” it does NOT auto-load AGENTS.md
41
57
  // (anthropics/claude-code#34235 is open; AGENTS.md works solely via an
@@ -0,0 +1,94 @@
1
+ /**
2
+ * Codex EVAL-tier transport β€” the runner + trace parser that
3
+ * `measureTriggerRate`/`runEval` dispatch to via the `ModelOutputParser` seam.
4
+ *
5
+ * SCHEMA: CONFIRMED against real `codex exec --json` (codex-cli 0.139.0, ChatGPT
6
+ * auth). The stream is the thread/item model:
7
+ *
8
+ * {"type":"thread.started","thread_id":"…"}
9
+ * {"type":"turn.started"}
10
+ * {"type":"item.started","item":{"id":"item_0","type":"command_execution",…}} // mid-flight
11
+ * {"type":"item.completed","item":{"id":"item_0","type":"command_execution","command":"…","aggregated_output":"…","exit_code":0}}
12
+ * {"type":"item.completed","item":{"id":"item_1","type":"agent_message","text":"…"}}
13
+ * {"type":"turn.completed","usage":{"input_tokens":…,"cached_input_tokens":…,"output_tokens":…}}
14
+ *
15
+ * So: assistant text = `item.completed` with `item.type:"agent_message"` β†’
16
+ * `item.text`; a tool call = `item.type:"command_execution"` β†’ `item.command`;
17
+ * usage rides `turn.completed`. We count `item.completed` ONLY (an `item.started`
18
+ * carries the same `id` mid-flight β€” counting both double-counts).
19
+ *
20
+ * THE SKILL FINDING: Codex has NO discrete "skill selected" event (its CLI has no
21
+ * Skill-tool concept). When a skill triggers, the model READS the skill's
22
+ * `SKILL.md` via a `command_execution` (`sed/cat … skills/<name>/SKILL.md`) and
23
+ * usually says so in an `agent_message`. So "did skill X fire" on Codex is not a
24
+ * clean trace event like Claude's `Skill` tool_use β€” it's detected by the
25
+ * SKILL.md read (`codexSkillFired`). Best-effort by nature (a cached skill might
26
+ * not be re-read); pair with a behavioral/judged check for certainty.
27
+ */
28
+ import type { ParsedModelRun, AgentRunArgs, RunOut, EvalDriver } from "../../eval.js";
29
+ import type { ToolCall } from "../../core/harness-driver.js";
30
+ /** Parse `codex exec --json` stdout into the common trace fields (confirmed schema). */
31
+ export declare function parseCodexEvalRun(out: {
32
+ stdout: string;
33
+ }): ParsedModelRun;
34
+ /**
35
+ * The error message if the run errored or was rate-limited (an `error` /
36
+ * `turn.failed` event), else null. CRITICAL for the eval tier: an errored turn
37
+ * must NOT be scored as a clean "skill didn't fire" miss β€” dogfooding hit a Codex
38
+ * usage limit ("You've hit your usage limit…") whose `error` event left an empty
39
+ * trace that `codexSkillFired` read as recall 0. A caller should skip/retry an
40
+ * errored run, not count it. (The Claude path has `isRateLimited` + backoff; this
41
+ * is the Codex equivalent detector.)
42
+ */
43
+ export declare function codexRunError(out: {
44
+ stdout: string;
45
+ }): string | null;
46
+ /**
47
+ * Did Codex activate skill `name` on this run? Detected by the SKILL.md read β€”
48
+ * Codex has no discrete skill-selection event, so when a skill triggers the model
49
+ * reads its `…/<name>/SKILL.md` via a `command_execution`. Best-effort (a cached
50
+ * skill might not be re-read); for the trigger-rate `fired` predicate over Codex.
51
+ */
52
+ export declare function codexSkillFired(run: {
53
+ toolCalls: readonly ToolCall[];
54
+ }, name: string): boolean;
55
+ /**
56
+ * Materialize a (Claude-shaped) plugin dir's skills into `<cwd>/.codex/skills/` β€”
57
+ * where Codex actually discovers them (validated live: codex reads
58
+ * `<cwd>/.codex/skills/<name>/SKILL.md`). This is the Codex analog of Claude's
59
+ * `--plugin-dir`: `measureTriggerRate` hands the runner a `pluginDir` (the
60
+ * stubbed/packaged skills), and the Codex runner installs them here before the
61
+ * turn. Pure fs β€” unit-testable without a binary.
62
+ */
63
+ export declare function installCodexSkills(pluginDir: string, cwd: string): number;
64
+ /**
65
+ * The Codex eval-tier `AgentRunner`: install the run's skills into `.codex/skills`
66
+ * (Codex's discovery path, vs Claude's `--plugin-dir`), then drive a real
67
+ * `codex exec --json` turn. The seam `measureTriggerRate(spec, { evalDriver:
68
+ * codexEvalDriver })` dispatches through.
69
+ */
70
+ export declare function codexEvalAgentRunner(args: AgentRunArgs): Promise<RunOut>;
71
+ /**
72
+ * The Codex eval driver β€” pass to `measureTriggerRate(spec, { evalDriver:
73
+ * codexEvalDriver })` to run a trigger-rate eval natively on `codex exec`. Pair
74
+ * the spec's `fired` with `codexSkillFired` (Codex has no Skill-tool event).
75
+ */
76
+ export declare const codexEvalDriver: EvalDriver;
77
+ /**
78
+ * Spawn real `codex exec --json` for the eval tier (real model, the user's codex
79
+ * auth β€” NOT the mock). CONFIRMED flags (codex 0.139.0): `--json` for the event
80
+ * stream, `--skip-git-repo-check` for a bare cwd, the approvals/sandbox bypass so
81
+ * the turn runs unattended, `-C <cwd>` for the working dir, prompt as the trailing
82
+ * positional, and stdin = /dev/null (`stdio: ["ignore",…]`) β€” codex otherwise
83
+ * blocks on "Reading additional input from stdin…". Needs ChatGPT/API auth +
84
+ * network egress to the model backend.
85
+ */
86
+ export declare function codexEvalRunner(args: {
87
+ task: string;
88
+ cwd: string;
89
+ timeoutMs: number;
90
+ }): {
91
+ code: number;
92
+ stdout: string;
93
+ };
94
+ //# sourceMappingURL=eval.d.ts.map
@@ -0,0 +1,227 @@
1
+ "use strict";
2
+ /**
3
+ * Codex EVAL-tier transport β€” the runner + trace parser that
4
+ * `measureTriggerRate`/`runEval` dispatch to via the `ModelOutputParser` seam.
5
+ *
6
+ * SCHEMA: CONFIRMED against real `codex exec --json` (codex-cli 0.139.0, ChatGPT
7
+ * auth). The stream is the thread/item model:
8
+ *
9
+ * {"type":"thread.started","thread_id":"…"}
10
+ * {"type":"turn.started"}
11
+ * {"type":"item.started","item":{"id":"item_0","type":"command_execution",…}} // mid-flight
12
+ * {"type":"item.completed","item":{"id":"item_0","type":"command_execution","command":"…","aggregated_output":"…","exit_code":0}}
13
+ * {"type":"item.completed","item":{"id":"item_1","type":"agent_message","text":"…"}}
14
+ * {"type":"turn.completed","usage":{"input_tokens":…,"cached_input_tokens":…,"output_tokens":…}}
15
+ *
16
+ * So: assistant text = `item.completed` with `item.type:"agent_message"` β†’
17
+ * `item.text`; a tool call = `item.type:"command_execution"` β†’ `item.command`;
18
+ * usage rides `turn.completed`. We count `item.completed` ONLY (an `item.started`
19
+ * carries the same `id` mid-flight β€” counting both double-counts).
20
+ *
21
+ * THE SKILL FINDING: Codex has NO discrete "skill selected" event (its CLI has no
22
+ * Skill-tool concept). When a skill triggers, the model READS the skill's
23
+ * `SKILL.md` via a `command_execution` (`sed/cat … skills/<name>/SKILL.md`) and
24
+ * usually says so in an `agent_message`. So "did skill X fire" on Codex is not a
25
+ * clean trace event like Claude's `Skill` tool_use β€” it's detected by the
26
+ * SKILL.md read (`codexSkillFired`). Best-effort by nature (a cached skill might
27
+ * not be re-read); pair with a behavioral/judged check for certainty.
28
+ */
29
+ Object.defineProperty(exports, "__esModule", { value: true });
30
+ exports.codexEvalDriver = void 0;
31
+ exports.parseCodexEvalRun = parseCodexEvalRun;
32
+ exports.codexRunError = codexRunError;
33
+ exports.codexSkillFired = codexSkillFired;
34
+ exports.installCodexSkills = installCodexSkills;
35
+ exports.codexEvalAgentRunner = codexEvalAgentRunner;
36
+ exports.codexEvalRunner = codexEvalRunner;
37
+ const node_child_process_1 = require("node:child_process");
38
+ const node_fs_1 = require("node:fs");
39
+ const node_path_1 = require("node:path");
40
+ const str = (v) => (typeof v === "string" ? v : "");
41
+ const num = (v) => (typeof v === "number" ? v : 0);
42
+ /** Parse the JSONL stream, skipping blank / non-JSON / malformed lines. */
43
+ function parseLines(stdout) {
44
+ const out = [];
45
+ for (const line of stdout.split(/\r?\n/)) {
46
+ const s = line.trim();
47
+ if (!s.startsWith("{"))
48
+ continue;
49
+ try {
50
+ out.push(JSON.parse(s));
51
+ }
52
+ catch {
53
+ /* tolerate a partial / non-event line */
54
+ }
55
+ }
56
+ return out;
57
+ }
58
+ /** Is this completed item a tool/command call (vs an agent_message / error)? */
59
+ function isToolItem(itemType) {
60
+ return (itemType === "command_execution" || /function_call|tool/.test(itemType));
61
+ }
62
+ function buildCall(item, itemType) {
63
+ return {
64
+ // command_execution β†’ the shell command; function-style β†’ its name.
65
+ name: str(item.command) || str(item.name) || itemType,
66
+ input: item,
67
+ resultText: str(item.aggregated_output),
68
+ isError: num(item.exit_code) !== 0 && item.exit_code != null,
69
+ };
70
+ }
71
+ /** Map a `turn.completed` usage block to the common EvalUsage. */
72
+ function usageFrom(u) {
73
+ return {
74
+ costUsd: 0, // codex on the ChatGPT sub reports no per-run USD
75
+ durationMs: 0,
76
+ inputTokens: num(u.input_tokens),
77
+ outputTokens: num(u.output_tokens),
78
+ cacheCreationTokens: 0,
79
+ cacheReadTokens: num(u.cached_input_tokens),
80
+ };
81
+ }
82
+ const ZERO_USAGE = {
83
+ costUsd: 0,
84
+ durationMs: 0,
85
+ inputTokens: 0,
86
+ outputTokens: 0,
87
+ cacheCreationTokens: 0,
88
+ cacheReadTokens: 0,
89
+ };
90
+ /** Parse `codex exec --json` stdout into the common trace fields (confirmed schema). */
91
+ function parseCodexEvalRun(out) {
92
+ const texts = [];
93
+ const toolCalls = [];
94
+ let usage = ZERO_USAGE;
95
+ for (const e of parseLines(out.stdout)) {
96
+ if (e.type === "turn.completed" && e.usage)
97
+ usage = usageFrom(e.usage);
98
+ // Count COMPLETED items only β€” item.started carries the same id mid-flight.
99
+ if (e.type !== "item.completed" || !e.item)
100
+ continue;
101
+ const itemType = str(e.item.type);
102
+ if (itemType === "agent_message") {
103
+ const t = str(e.item.text);
104
+ if (t)
105
+ texts.push(t);
106
+ }
107
+ else if (isToolItem(itemType)) {
108
+ toolCalls.push(buildCall(e.item, itemType));
109
+ }
110
+ }
111
+ return {
112
+ // Fallback to trimmed stdout if no agent_message was seen (keeps output non-empty).
113
+ output: texts.join("\n") || out.stdout.trim(),
114
+ turns: texts.length,
115
+ toolCalls,
116
+ hooks: [],
117
+ subagents: [],
118
+ usage,
119
+ };
120
+ }
121
+ /**
122
+ * The error message if the run errored or was rate-limited (an `error` /
123
+ * `turn.failed` event), else null. CRITICAL for the eval tier: an errored turn
124
+ * must NOT be scored as a clean "skill didn't fire" miss β€” dogfooding hit a Codex
125
+ * usage limit ("You've hit your usage limit…") whose `error` event left an empty
126
+ * trace that `codexSkillFired` read as recall 0. A caller should skip/retry an
127
+ * errored run, not count it. (The Claude path has `isRateLimited` + backoff; this
128
+ * is the Codex equivalent detector.)
129
+ */
130
+ function codexRunError(out) {
131
+ for (const e of parseLines(out.stdout)) {
132
+ if (e.type === "error")
133
+ return str(e.message) || "codex error";
134
+ if (e.type === "turn.failed")
135
+ return str(e.error?.message) || "turn failed";
136
+ }
137
+ return null;
138
+ }
139
+ /**
140
+ * Did Codex activate skill `name` on this run? Detected by the SKILL.md read β€”
141
+ * Codex has no discrete skill-selection event, so when a skill triggers the model
142
+ * reads its `…/<name>/SKILL.md` via a `command_execution`. Best-effort (a cached
143
+ * skill might not be re-read); for the trigger-rate `fired` predicate over Codex.
144
+ */
145
+ function codexSkillFired(run, name) {
146
+ const needle = `${name}/SKILL.md`;
147
+ return run.toolCalls.some((c) => str(c.name).includes(needle));
148
+ }
149
+ /**
150
+ * Materialize a (Claude-shaped) plugin dir's skills into `<cwd>/.codex/skills/` β€”
151
+ * where Codex actually discovers them (validated live: codex reads
152
+ * `<cwd>/.codex/skills/<name>/SKILL.md`). This is the Codex analog of Claude's
153
+ * `--plugin-dir`: `measureTriggerRate` hands the runner a `pluginDir` (the
154
+ * stubbed/packaged skills), and the Codex runner installs them here before the
155
+ * turn. Pure fs β€” unit-testable without a binary.
156
+ */
157
+ function installCodexSkills(pluginDir, cwd) {
158
+ const skillsRoot = (0, node_path_1.join)(pluginDir, "skills");
159
+ if (!(0, node_fs_1.existsSync)(skillsRoot))
160
+ return 0;
161
+ let n = 0;
162
+ for (const name of (0, node_fs_1.readdirSync)(skillsRoot)) {
163
+ const src = (0, node_path_1.join)(skillsRoot, name, "SKILL.md");
164
+ if (!(0, node_fs_1.existsSync)(src))
165
+ continue;
166
+ const dest = (0, node_path_1.join)(cwd, ".codex", "skills", name, "SKILL.md");
167
+ (0, node_fs_1.mkdirSync)((0, node_path_1.dirname)(dest), { recursive: true });
168
+ (0, node_fs_1.writeFileSync)(dest, (0, node_fs_1.readFileSync)(src, "utf-8"));
169
+ n += 1;
170
+ }
171
+ return n;
172
+ }
173
+ /* v8 ignore start -- real codex subprocess; validated against the binary, not the unit gate */
174
+ /**
175
+ * The Codex eval-tier `AgentRunner`: install the run's skills into `.codex/skills`
176
+ * (Codex's discovery path, vs Claude's `--plugin-dir`), then drive a real
177
+ * `codex exec --json` turn. The seam `measureTriggerRate(spec, { evalDriver:
178
+ * codexEvalDriver })` dispatches through.
179
+ */
180
+ function codexEvalAgentRunner(args) {
181
+ if (args.pluginDir)
182
+ installCodexSkills(args.pluginDir, args.cwd);
183
+ return Promise.resolve(codexEvalRunner({
184
+ task: args.task,
185
+ cwd: args.cwd,
186
+ timeoutMs: args.timeoutMs,
187
+ }));
188
+ }
189
+ /**
190
+ * The Codex eval driver β€” pass to `measureTriggerRate(spec, { evalDriver:
191
+ * codexEvalDriver })` to run a trigger-rate eval natively on `codex exec`. Pair
192
+ * the spec's `fired` with `codexSkillFired` (Codex has no Skill-tool event).
193
+ */
194
+ exports.codexEvalDriver = {
195
+ runner: codexEvalAgentRunner,
196
+ parse: parseCodexEvalRun,
197
+ runError: codexRunError,
198
+ };
199
+ /**
200
+ * Spawn real `codex exec --json` for the eval tier (real model, the user's codex
201
+ * auth β€” NOT the mock). CONFIRMED flags (codex 0.139.0): `--json` for the event
202
+ * stream, `--skip-git-repo-check` for a bare cwd, the approvals/sandbox bypass so
203
+ * the turn runs unattended, `-C <cwd>` for the working dir, prompt as the trailing
204
+ * positional, and stdin = /dev/null (`stdio: ["ignore",…]`) β€” codex otherwise
205
+ * blocks on "Reading additional input from stdin…". Needs ChatGPT/API auth +
206
+ * network egress to the model backend.
207
+ */
208
+ function codexEvalRunner(args) {
209
+ const r = (0, node_child_process_1.spawnSync)("codex", [
210
+ "exec",
211
+ "--json",
212
+ "--skip-git-repo-check",
213
+ "--dangerously-bypass-approvals-and-sandbox",
214
+ "-C",
215
+ args.cwd,
216
+ args.task,
217
+ ], {
218
+ cwd: args.cwd,
219
+ encoding: "utf-8",
220
+ timeout: args.timeoutMs,
221
+ stdio: ["ignore", "pipe", "pipe"],
222
+ maxBuffer: 64 * 1024 * 1024,
223
+ });
224
+ return { code: r.status ?? 1, stdout: r.stdout ?? "" };
225
+ }
226
+ /* v8 ignore stop */
227
+ //# sourceMappingURL=eval.js.map