vigiles 5.0.1 β 5.1.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +14 -8
- package/dist/adapters/claude-code/agent-runtime.d.ts +10 -0
- package/dist/adapters/claude-code/agent-runtime.js +15 -29
- package/dist/adapters/claude-code/dialect.js +18 -2
- package/dist/adapters/codex/eval.d.ts +94 -0
- package/dist/adapters/codex/eval.js +227 -0
- package/dist/cli.js +464 -8
- package/dist/codex.d.ts +1 -0
- package/dist/codex.js +3 -0
- package/dist/core/compile.js +8 -36
- package/dist/core/description-overlap.d.ts +27 -0
- package/dist/core/description-overlap.js +53 -0
- package/dist/core/dialect.d.ts +8 -0
- package/dist/core/frontmatter-read.d.ts +25 -0
- package/dist/core/frontmatter-read.js +138 -0
- package/dist/core/hook-events.d.ts +34 -0
- package/dist/core/hook-events.js +48 -0
- package/dist/core/mcp-config.d.ts +20 -0
- package/dist/core/mcp-config.js +40 -0
- package/dist/core/mcp-hook.d.ts +35 -0
- package/dist/core/mcp-hook.js +70 -0
- package/dist/core/mcp-tool.d.ts +50 -0
- package/dist/core/mcp-tool.js +61 -0
- package/dist/core/tool-contract.d.ts +68 -0
- package/dist/core/tool-contract.js +113 -0
- package/dist/core/types.d.ts +89 -0
- package/dist/core/validate.js +22 -0
- package/dist/eval.d.ts +69 -13
- package/dist/eval.js +106 -51
- package/dist/leaderboard.js +61 -3
- package/dist/plugin-loader.d.ts +1 -0
- package/dist/plugin-loader.js +71 -18
- package/dist/scan-behavioral.d.ts +73 -0
- package/dist/scan-behavioral.js +150 -0
- package/dist/scan.d.ts +126 -1
- package/dist/scan.js +559 -40
- package/package.json +1 -1
- package/skills/migrate-to-spec/SKILL.md +0 -2
package/README.md
CHANGED
|
@@ -30,19 +30,18 @@ deterministic layer for it, and does two independent things β adopt either, or
|
|
|
30
30
|
| **π§ͺ Test** | Hooks, skills, and subagents are code. vigiles tests they _do their job_ β and almost all of it is **deterministic, no API key**; the real-model evals run on your **Claude subscription**, not metered tokens. **[β](docs/harness-testing.md)** |
|
|
31
31
|
|
|
32
32
|
Pick the one that hurts today. **Works with Claude Code and Codex**
|
|
33
|
-
([`vigiles/codex`](docs/harnesses.md))
|
|
34
|
-
[
|
|
33
|
+
([`vigiles/codex`](docs/harnesses.md)), and you can
|
|
34
|
+
[teach it your own harness](docs/authoring-an-adapter.md).
|
|
35
35
|
|
|
36
36
|
## Quick start
|
|
37
37
|
|
|
38
38
|
**Paste into Claude Code or Codex:**
|
|
39
39
|
|
|
40
40
|
```text
|
|
41
|
-
|
|
42
|
-
references and show me what's stale, then write
|
|
43
|
-
of my hooks or skills.
|
|
44
|
-
|
|
45
|
-
whether to enforce strictly (--strict).
|
|
41
|
+
Set up vigiles in this repo with good defaults (lint + test, non-interactive).
|
|
42
|
+
Verify my CLAUDE.md / AGENTS.md references and show me what's stale, then write
|
|
43
|
+
and run a harness test for one of my hooks or skills. Ask me first before gating
|
|
44
|
+
it in CI, adding a real-model eval, or enforcing strictly (--strict).
|
|
46
45
|
```
|
|
47
46
|
|
|
48
47
|
Or do it yourself:
|
|
@@ -95,6 +94,12 @@ File paths, scripts, and code symbols β plus linter rules across **7 catalogs*
|
|
|
95
94
|
step up to a typed `.spec.ts` (compiled to CLAUDE.md, compiler-grade) when you want
|
|
96
95
|
it. **[Full guide β](docs/verifying-instruction-files.md)**
|
|
97
96
|
|
|
97
|
+
**Same cross-reference, any plugin.** `npx vigiles scan` checks a plugin's
|
|
98
|
+
_contracts_ β every subagent tool, `mcp__server__tool`, `mcp_tool` hook, hook
|
|
99
|
+
event, and script path actually **exists and resolves**, not just parses (valid
|
|
100
|
+
YAML β a tool that's real). A **superset of Anthropic's `claude plugin validate`**,
|
|
101
|
+
no key. **[Audit any plugin β](docs/cli.md#scan-dir)**
|
|
102
|
+
|
|
98
103
|
## β‘ Test β does your harness do its job?
|
|
99
104
|
|
|
100
105
|
A hook can be wired wrong. A skill's description can fail to trigger β or hijack
|
|
@@ -132,7 +137,8 @@ That's why you can eval your harness on every change, not just once.
|
|
|
132
137
|
|
|
133
138
|
## More
|
|
134
139
|
|
|
135
|
-
- **[
|
|
140
|
+
- **[Plugin health leaderboard β](docs/cli.md#scan-dir)** β point `scan` at a marketplace (e.g. `wshobson/agents`) and it ranks every plugin by structural health (0β100, AβF), worst issues first β still **no key**. Add `--trigger` for the model-gated column: do the skills actually fire?
|
|
141
|
+
- **[CLI & GitHub Action β](docs/cli.md)** β every command, the Action (inputs / output / versioning), the Claude Code plugin, and the `lint` rules.
|
|
136
142
|
- **[Skills β](docs/skills.md)** β consumer skills installed as a Claude Code plugin: `/plugin marketplace add zernie/vigiles` then `/plugin install vigiles@vigiles` (or let `vigiles init` do it). The model-invocable ones (`test-harness`, `strengthen`, `edit-spec`) fire on their own β ask _"test my skills"_, _"strengthen my rules"_, or _"add a rule to CLAUDE.md"_ and the agent reaches for them; `migrate-to-spec` and `linter-docs` are user-invoked.
|
|
137
143
|
- **[Docs index β](docs/README.md)** Β· **[Research β](research/README.md)** Β· **[Related tools β](docs/related-tools.md)** (ast-grep, Dependency Cruiser, Ruler, rulesync).
|
|
138
144
|
- Companion to [Feedback Loop Is All You Need](https://zernie.com/blog/feedback-loop-is-all-you-need).
|
|
@@ -31,6 +31,16 @@
|
|
|
31
31
|
* an empty list (`tools:` with nothing after it) means "no tools allowed".
|
|
32
32
|
*/
|
|
33
33
|
export declare function parseAgentTools(markdown: string): string[] | null;
|
|
34
|
+
/**
|
|
35
|
+
* Parse a comma/array tool list under an arbitrary frontmatter `key` (e.g.
|
|
36
|
+
* `tools:` or `disallowedTools:`) via the shared lenient reader
|
|
37
|
+
* (core/frontmatter-read.ts): a real YAML parse (so `key: [Read, "Bash"]` is a
|
|
38
|
+
* native array and `key: Read, Bash` a comma scalar) with a regex salvage when
|
|
39
|
+
* the block is malformed β the rail still reads the contract. `null` when the key
|
|
40
|
+
* is absent (inherits all), `[]` when present-but-empty (no tools). Shared by the
|
|
41
|
+
* rail (`tools:`) and the `disallowed-tools-contract` scan/lint.
|
|
42
|
+
*/
|
|
43
|
+
export declare function parseAgentToolList(markdown: string, key: string): string[] | null;
|
|
34
44
|
export interface PreToolDecision {
|
|
35
45
|
/** Whether the tool call is allowed (true) or blocked (false). */
|
|
36
46
|
readonly allow: boolean;
|
|
@@ -25,6 +25,7 @@
|
|
|
25
25
|
*/
|
|
26
26
|
Object.defineProperty(exports, "__esModule", { value: true });
|
|
27
27
|
exports.parseAgentTools = parseAgentTools;
|
|
28
|
+
exports.parseAgentToolList = parseAgentToolList;
|
|
28
29
|
exports.decidePreToolUse = decidePreToolUse;
|
|
29
30
|
exports.setActiveAgent = setActiveAgent;
|
|
30
31
|
exports.clearActiveAgent = clearActiveAgent;
|
|
@@ -32,28 +33,10 @@ exports.readActiveAgent = readActiveAgent;
|
|
|
32
33
|
exports.evaluatePreToolUse = evaluatePreToolUse;
|
|
33
34
|
const node_fs_1 = require("node:fs");
|
|
34
35
|
const node_path_1 = require("node:path");
|
|
36
|
+
const frontmatter_read_js_1 = require("../../core/frontmatter-read.js");
|
|
35
37
|
// ---------------------------------------------------------------------------
|
|
36
38
|
// Parse the tool contract from a compiled agent .md
|
|
37
39
|
// ---------------------------------------------------------------------------
|
|
38
|
-
/** Extract the YAML frontmatter block (between the first pair of `---` fences). */
|
|
39
|
-
function extractFrontmatter(markdown) {
|
|
40
|
-
const lines = markdown.split("\n");
|
|
41
|
-
let start = -1;
|
|
42
|
-
for (let i = 0; i < lines.length; i++) {
|
|
43
|
-
if (lines[i].trim() === "---") {
|
|
44
|
-
start = i;
|
|
45
|
-
break;
|
|
46
|
-
}
|
|
47
|
-
}
|
|
48
|
-
if (start === -1)
|
|
49
|
-
return null;
|
|
50
|
-
for (let i = start + 1; i < lines.length; i++) {
|
|
51
|
-
if (lines[i].trim() === "---") {
|
|
52
|
-
return lines.slice(start + 1, i).join("\n");
|
|
53
|
-
}
|
|
54
|
-
}
|
|
55
|
-
return null;
|
|
56
|
-
}
|
|
57
40
|
/**
|
|
58
41
|
* Parse an agent's allowed-tools contract from its compiled markdown.
|
|
59
42
|
*
|
|
@@ -63,16 +46,19 @@ function extractFrontmatter(markdown) {
|
|
|
63
46
|
* an empty list (`tools:` with nothing after it) means "no tools allowed".
|
|
64
47
|
*/
|
|
65
48
|
function parseAgentTools(markdown) {
|
|
66
|
-
|
|
67
|
-
|
|
68
|
-
|
|
69
|
-
|
|
70
|
-
|
|
71
|
-
|
|
72
|
-
|
|
73
|
-
|
|
74
|
-
|
|
75
|
-
|
|
49
|
+
return parseAgentToolList(markdown, "tools");
|
|
50
|
+
}
|
|
51
|
+
/**
|
|
52
|
+
* Parse a comma/array tool list under an arbitrary frontmatter `key` (e.g.
|
|
53
|
+
* `tools:` or `disallowedTools:`) via the shared lenient reader
|
|
54
|
+
* (core/frontmatter-read.ts): a real YAML parse (so `key: [Read, "Bash"]` is a
|
|
55
|
+
* native array and `key: Read, Bash` a comma scalar) with a regex salvage when
|
|
56
|
+
* the block is malformed β the rail still reads the contract. `null` when the key
|
|
57
|
+
* is absent (inherits all), `[]` when present-but-empty (no tools). Shared by the
|
|
58
|
+
* rail (`tools:`) and the `disallowed-tools-contract` scan/lint.
|
|
59
|
+
*/
|
|
60
|
+
function parseAgentToolList(markdown, key) {
|
|
61
|
+
return (0, frontmatter_read_js_1.frontmatterList)((0, frontmatter_read_js_1.readFrontmatter)(markdown), key);
|
|
76
62
|
}
|
|
77
63
|
/**
|
|
78
64
|
* Decide whether `tool` is allowed under an agent's tool contract. Pure, so the
|
|
@@ -9,9 +9,13 @@ exports.claudeCodeDialect = {
|
|
|
9
9
|
"Read",
|
|
10
10
|
"Write",
|
|
11
11
|
"Edit",
|
|
12
|
+
"MultiEdit",
|
|
12
13
|
"Bash",
|
|
14
|
+
"BashOutput",
|
|
15
|
+
"KillBash",
|
|
13
16
|
"Grep",
|
|
14
17
|
"Glob",
|
|
18
|
+
"LS",
|
|
15
19
|
"WebSearch",
|
|
16
20
|
"WebFetch",
|
|
17
21
|
"NotebookEdit",
|
|
@@ -30,12 +34,24 @@ exports.claudeCodeDialect = {
|
|
|
30
34
|
"WaitForMcpServers",
|
|
31
35
|
],
|
|
32
36
|
mcpToolPattern: /^mcp__[a-z0-9_-]+__[a-z0-9_-]+$/i,
|
|
37
|
+
// Claude Code's own built-in MCP server: the IDE integration provides
|
|
38
|
+
// `mcp__ide__getDiagnostics` / `mcp__ide__executeCode` at runtime without any
|
|
39
|
+
// plugin declaring it, so a contract that lists those must NOT be flagged as
|
|
40
|
+
// referencing an undeclared server (the mcp-tool-resolves allowlist).
|
|
41
|
+
knownMcpServers: ["ide"],
|
|
42
|
+
// The real Claude Code hook events. (Was wrong: PreSession/PostSession don't
|
|
43
|
+
// exist; SessionStart/SessionEnd/Stop/SubagentStop/UserPromptSubmit/PreCompact
|
|
44
|
+
// were missing β verified against the events real plugins register.)
|
|
33
45
|
hookEvents: [
|
|
34
46
|
"PreToolUse",
|
|
35
47
|
"PostToolUse",
|
|
36
|
-
"
|
|
37
|
-
"PostSession",
|
|
48
|
+
"UserPromptSubmit",
|
|
38
49
|
"Notification",
|
|
50
|
+
"Stop",
|
|
51
|
+
"SubagentStop",
|
|
52
|
+
"PreCompact",
|
|
53
|
+
"SessionStart",
|
|
54
|
+
"SessionEnd",
|
|
39
55
|
],
|
|
40
56
|
// Claude Code natively reads CLAUDE.md only β it does NOT auto-load AGENTS.md
|
|
41
57
|
// (anthropics/claude-code#34235 is open; AGENTS.md works solely via an
|
|
@@ -0,0 +1,94 @@
|
|
|
1
|
+
/**
|
|
2
|
+
* Codex EVAL-tier transport β the runner + trace parser that
|
|
3
|
+
* `measureTriggerRate`/`runEval` dispatch to via the `ModelOutputParser` seam.
|
|
4
|
+
*
|
|
5
|
+
* SCHEMA: CONFIRMED against real `codex exec --json` (codex-cli 0.139.0, ChatGPT
|
|
6
|
+
* auth). The stream is the thread/item model:
|
|
7
|
+
*
|
|
8
|
+
* {"type":"thread.started","thread_id":"β¦"}
|
|
9
|
+
* {"type":"turn.started"}
|
|
10
|
+
* {"type":"item.started","item":{"id":"item_0","type":"command_execution",β¦}} // mid-flight
|
|
11
|
+
* {"type":"item.completed","item":{"id":"item_0","type":"command_execution","command":"β¦","aggregated_output":"β¦","exit_code":0}}
|
|
12
|
+
* {"type":"item.completed","item":{"id":"item_1","type":"agent_message","text":"β¦"}}
|
|
13
|
+
* {"type":"turn.completed","usage":{"input_tokens":β¦,"cached_input_tokens":β¦,"output_tokens":β¦}}
|
|
14
|
+
*
|
|
15
|
+
* So: assistant text = `item.completed` with `item.type:"agent_message"` β
|
|
16
|
+
* `item.text`; a tool call = `item.type:"command_execution"` β `item.command`;
|
|
17
|
+
* usage rides `turn.completed`. We count `item.completed` ONLY (an `item.started`
|
|
18
|
+
* carries the same `id` mid-flight β counting both double-counts).
|
|
19
|
+
*
|
|
20
|
+
* THE SKILL FINDING: Codex has NO discrete "skill selected" event (its CLI has no
|
|
21
|
+
* Skill-tool concept). When a skill triggers, the model READS the skill's
|
|
22
|
+
* `SKILL.md` via a `command_execution` (`sed/cat β¦ skills/<name>/SKILL.md`) and
|
|
23
|
+
* usually says so in an `agent_message`. So "did skill X fire" on Codex is not a
|
|
24
|
+
* clean trace event like Claude's `Skill` tool_use β it's detected by the
|
|
25
|
+
* SKILL.md read (`codexSkillFired`). Best-effort by nature (a cached skill might
|
|
26
|
+
* not be re-read); pair with a behavioral/judged check for certainty.
|
|
27
|
+
*/
|
|
28
|
+
import type { ParsedModelRun, AgentRunArgs, RunOut, EvalDriver } from "../../eval.js";
|
|
29
|
+
import type { ToolCall } from "../../core/harness-driver.js";
|
|
30
|
+
/** Parse `codex exec --json` stdout into the common trace fields (confirmed schema). */
|
|
31
|
+
export declare function parseCodexEvalRun(out: {
|
|
32
|
+
stdout: string;
|
|
33
|
+
}): ParsedModelRun;
|
|
34
|
+
/**
|
|
35
|
+
* The error message if the run errored or was rate-limited (an `error` /
|
|
36
|
+
* `turn.failed` event), else null. CRITICAL for the eval tier: an errored turn
|
|
37
|
+
* must NOT be scored as a clean "skill didn't fire" miss β dogfooding hit a Codex
|
|
38
|
+
* usage limit ("You've hit your usage limitβ¦") whose `error` event left an empty
|
|
39
|
+
* trace that `codexSkillFired` read as recall 0. A caller should skip/retry an
|
|
40
|
+
* errored run, not count it. (The Claude path has `isRateLimited` + backoff; this
|
|
41
|
+
* is the Codex equivalent detector.)
|
|
42
|
+
*/
|
|
43
|
+
export declare function codexRunError(out: {
|
|
44
|
+
stdout: string;
|
|
45
|
+
}): string | null;
|
|
46
|
+
/**
|
|
47
|
+
* Did Codex activate skill `name` on this run? Detected by the SKILL.md read β
|
|
48
|
+
* Codex has no discrete skill-selection event, so when a skill triggers the model
|
|
49
|
+
* reads its `β¦/<name>/SKILL.md` via a `command_execution`. Best-effort (a cached
|
|
50
|
+
* skill might not be re-read); for the trigger-rate `fired` predicate over Codex.
|
|
51
|
+
*/
|
|
52
|
+
export declare function codexSkillFired(run: {
|
|
53
|
+
toolCalls: readonly ToolCall[];
|
|
54
|
+
}, name: string): boolean;
|
|
55
|
+
/**
|
|
56
|
+
* Materialize a (Claude-shaped) plugin dir's skills into `<cwd>/.codex/skills/` β
|
|
57
|
+
* where Codex actually discovers them (validated live: codex reads
|
|
58
|
+
* `<cwd>/.codex/skills/<name>/SKILL.md`). This is the Codex analog of Claude's
|
|
59
|
+
* `--plugin-dir`: `measureTriggerRate` hands the runner a `pluginDir` (the
|
|
60
|
+
* stubbed/packaged skills), and the Codex runner installs them here before the
|
|
61
|
+
* turn. Pure fs β unit-testable without a binary.
|
|
62
|
+
*/
|
|
63
|
+
export declare function installCodexSkills(pluginDir: string, cwd: string): number;
|
|
64
|
+
/**
|
|
65
|
+
* The Codex eval-tier `AgentRunner`: install the run's skills into `.codex/skills`
|
|
66
|
+
* (Codex's discovery path, vs Claude's `--plugin-dir`), then drive a real
|
|
67
|
+
* `codex exec --json` turn. The seam `measureTriggerRate(spec, { evalDriver:
|
|
68
|
+
* codexEvalDriver })` dispatches through.
|
|
69
|
+
*/
|
|
70
|
+
export declare function codexEvalAgentRunner(args: AgentRunArgs): Promise<RunOut>;
|
|
71
|
+
/**
|
|
72
|
+
* The Codex eval driver β pass to `measureTriggerRate(spec, { evalDriver:
|
|
73
|
+
* codexEvalDriver })` to run a trigger-rate eval natively on `codex exec`. Pair
|
|
74
|
+
* the spec's `fired` with `codexSkillFired` (Codex has no Skill-tool event).
|
|
75
|
+
*/
|
|
76
|
+
export declare const codexEvalDriver: EvalDriver;
|
|
77
|
+
/**
|
|
78
|
+
* Spawn real `codex exec --json` for the eval tier (real model, the user's codex
|
|
79
|
+
* auth β NOT the mock). CONFIRMED flags (codex 0.139.0): `--json` for the event
|
|
80
|
+
* stream, `--skip-git-repo-check` for a bare cwd, the approvals/sandbox bypass so
|
|
81
|
+
* the turn runs unattended, `-C <cwd>` for the working dir, prompt as the trailing
|
|
82
|
+
* positional, and stdin = /dev/null (`stdio: ["ignore",β¦]`) β codex otherwise
|
|
83
|
+
* blocks on "Reading additional input from stdinβ¦". Needs ChatGPT/API auth +
|
|
84
|
+
* network egress to the model backend.
|
|
85
|
+
*/
|
|
86
|
+
export declare function codexEvalRunner(args: {
|
|
87
|
+
task: string;
|
|
88
|
+
cwd: string;
|
|
89
|
+
timeoutMs: number;
|
|
90
|
+
}): {
|
|
91
|
+
code: number;
|
|
92
|
+
stdout: string;
|
|
93
|
+
};
|
|
94
|
+
//# sourceMappingURL=eval.d.ts.map
|
|
@@ -0,0 +1,227 @@
|
|
|
1
|
+
"use strict";
|
|
2
|
+
/**
|
|
3
|
+
* Codex EVAL-tier transport β the runner + trace parser that
|
|
4
|
+
* `measureTriggerRate`/`runEval` dispatch to via the `ModelOutputParser` seam.
|
|
5
|
+
*
|
|
6
|
+
* SCHEMA: CONFIRMED against real `codex exec --json` (codex-cli 0.139.0, ChatGPT
|
|
7
|
+
* auth). The stream is the thread/item model:
|
|
8
|
+
*
|
|
9
|
+
* {"type":"thread.started","thread_id":"β¦"}
|
|
10
|
+
* {"type":"turn.started"}
|
|
11
|
+
* {"type":"item.started","item":{"id":"item_0","type":"command_execution",β¦}} // mid-flight
|
|
12
|
+
* {"type":"item.completed","item":{"id":"item_0","type":"command_execution","command":"β¦","aggregated_output":"β¦","exit_code":0}}
|
|
13
|
+
* {"type":"item.completed","item":{"id":"item_1","type":"agent_message","text":"β¦"}}
|
|
14
|
+
* {"type":"turn.completed","usage":{"input_tokens":β¦,"cached_input_tokens":β¦,"output_tokens":β¦}}
|
|
15
|
+
*
|
|
16
|
+
* So: assistant text = `item.completed` with `item.type:"agent_message"` β
|
|
17
|
+
* `item.text`; a tool call = `item.type:"command_execution"` β `item.command`;
|
|
18
|
+
* usage rides `turn.completed`. We count `item.completed` ONLY (an `item.started`
|
|
19
|
+
* carries the same `id` mid-flight β counting both double-counts).
|
|
20
|
+
*
|
|
21
|
+
* THE SKILL FINDING: Codex has NO discrete "skill selected" event (its CLI has no
|
|
22
|
+
* Skill-tool concept). When a skill triggers, the model READS the skill's
|
|
23
|
+
* `SKILL.md` via a `command_execution` (`sed/cat β¦ skills/<name>/SKILL.md`) and
|
|
24
|
+
* usually says so in an `agent_message`. So "did skill X fire" on Codex is not a
|
|
25
|
+
* clean trace event like Claude's `Skill` tool_use β it's detected by the
|
|
26
|
+
* SKILL.md read (`codexSkillFired`). Best-effort by nature (a cached skill might
|
|
27
|
+
* not be re-read); pair with a behavioral/judged check for certainty.
|
|
28
|
+
*/
|
|
29
|
+
Object.defineProperty(exports, "__esModule", { value: true });
|
|
30
|
+
exports.codexEvalDriver = void 0;
|
|
31
|
+
exports.parseCodexEvalRun = parseCodexEvalRun;
|
|
32
|
+
exports.codexRunError = codexRunError;
|
|
33
|
+
exports.codexSkillFired = codexSkillFired;
|
|
34
|
+
exports.installCodexSkills = installCodexSkills;
|
|
35
|
+
exports.codexEvalAgentRunner = codexEvalAgentRunner;
|
|
36
|
+
exports.codexEvalRunner = codexEvalRunner;
|
|
37
|
+
const node_child_process_1 = require("node:child_process");
|
|
38
|
+
const node_fs_1 = require("node:fs");
|
|
39
|
+
const node_path_1 = require("node:path");
|
|
40
|
+
const str = (v) => (typeof v === "string" ? v : "");
|
|
41
|
+
const num = (v) => (typeof v === "number" ? v : 0);
|
|
42
|
+
/** Parse the JSONL stream, skipping blank / non-JSON / malformed lines. */
|
|
43
|
+
function parseLines(stdout) {
|
|
44
|
+
const out = [];
|
|
45
|
+
for (const line of stdout.split(/\r?\n/)) {
|
|
46
|
+
const s = line.trim();
|
|
47
|
+
if (!s.startsWith("{"))
|
|
48
|
+
continue;
|
|
49
|
+
try {
|
|
50
|
+
out.push(JSON.parse(s));
|
|
51
|
+
}
|
|
52
|
+
catch {
|
|
53
|
+
/* tolerate a partial / non-event line */
|
|
54
|
+
}
|
|
55
|
+
}
|
|
56
|
+
return out;
|
|
57
|
+
}
|
|
58
|
+
/** Is this completed item a tool/command call (vs an agent_message / error)? */
|
|
59
|
+
function isToolItem(itemType) {
|
|
60
|
+
return (itemType === "command_execution" || /function_call|tool/.test(itemType));
|
|
61
|
+
}
|
|
62
|
+
function buildCall(item, itemType) {
|
|
63
|
+
return {
|
|
64
|
+
// command_execution β the shell command; function-style β its name.
|
|
65
|
+
name: str(item.command) || str(item.name) || itemType,
|
|
66
|
+
input: item,
|
|
67
|
+
resultText: str(item.aggregated_output),
|
|
68
|
+
isError: num(item.exit_code) !== 0 && item.exit_code != null,
|
|
69
|
+
};
|
|
70
|
+
}
|
|
71
|
+
/** Map a `turn.completed` usage block to the common EvalUsage. */
|
|
72
|
+
function usageFrom(u) {
|
|
73
|
+
return {
|
|
74
|
+
costUsd: 0, // codex on the ChatGPT sub reports no per-run USD
|
|
75
|
+
durationMs: 0,
|
|
76
|
+
inputTokens: num(u.input_tokens),
|
|
77
|
+
outputTokens: num(u.output_tokens),
|
|
78
|
+
cacheCreationTokens: 0,
|
|
79
|
+
cacheReadTokens: num(u.cached_input_tokens),
|
|
80
|
+
};
|
|
81
|
+
}
|
|
82
|
+
const ZERO_USAGE = {
|
|
83
|
+
costUsd: 0,
|
|
84
|
+
durationMs: 0,
|
|
85
|
+
inputTokens: 0,
|
|
86
|
+
outputTokens: 0,
|
|
87
|
+
cacheCreationTokens: 0,
|
|
88
|
+
cacheReadTokens: 0,
|
|
89
|
+
};
|
|
90
|
+
/** Parse `codex exec --json` stdout into the common trace fields (confirmed schema). */
|
|
91
|
+
function parseCodexEvalRun(out) {
|
|
92
|
+
const texts = [];
|
|
93
|
+
const toolCalls = [];
|
|
94
|
+
let usage = ZERO_USAGE;
|
|
95
|
+
for (const e of parseLines(out.stdout)) {
|
|
96
|
+
if (e.type === "turn.completed" && e.usage)
|
|
97
|
+
usage = usageFrom(e.usage);
|
|
98
|
+
// Count COMPLETED items only β item.started carries the same id mid-flight.
|
|
99
|
+
if (e.type !== "item.completed" || !e.item)
|
|
100
|
+
continue;
|
|
101
|
+
const itemType = str(e.item.type);
|
|
102
|
+
if (itemType === "agent_message") {
|
|
103
|
+
const t = str(e.item.text);
|
|
104
|
+
if (t)
|
|
105
|
+
texts.push(t);
|
|
106
|
+
}
|
|
107
|
+
else if (isToolItem(itemType)) {
|
|
108
|
+
toolCalls.push(buildCall(e.item, itemType));
|
|
109
|
+
}
|
|
110
|
+
}
|
|
111
|
+
return {
|
|
112
|
+
// Fallback to trimmed stdout if no agent_message was seen (keeps output non-empty).
|
|
113
|
+
output: texts.join("\n") || out.stdout.trim(),
|
|
114
|
+
turns: texts.length,
|
|
115
|
+
toolCalls,
|
|
116
|
+
hooks: [],
|
|
117
|
+
subagents: [],
|
|
118
|
+
usage,
|
|
119
|
+
};
|
|
120
|
+
}
|
|
121
|
+
/**
|
|
122
|
+
* The error message if the run errored or was rate-limited (an `error` /
|
|
123
|
+
* `turn.failed` event), else null. CRITICAL for the eval tier: an errored turn
|
|
124
|
+
* must NOT be scored as a clean "skill didn't fire" miss β dogfooding hit a Codex
|
|
125
|
+
* usage limit ("You've hit your usage limitβ¦") whose `error` event left an empty
|
|
126
|
+
* trace that `codexSkillFired` read as recall 0. A caller should skip/retry an
|
|
127
|
+
* errored run, not count it. (The Claude path has `isRateLimited` + backoff; this
|
|
128
|
+
* is the Codex equivalent detector.)
|
|
129
|
+
*/
|
|
130
|
+
function codexRunError(out) {
|
|
131
|
+
for (const e of parseLines(out.stdout)) {
|
|
132
|
+
if (e.type === "error")
|
|
133
|
+
return str(e.message) || "codex error";
|
|
134
|
+
if (e.type === "turn.failed")
|
|
135
|
+
return str(e.error?.message) || "turn failed";
|
|
136
|
+
}
|
|
137
|
+
return null;
|
|
138
|
+
}
|
|
139
|
+
/**
|
|
140
|
+
* Did Codex activate skill `name` on this run? Detected by the SKILL.md read β
|
|
141
|
+
* Codex has no discrete skill-selection event, so when a skill triggers the model
|
|
142
|
+
* reads its `β¦/<name>/SKILL.md` via a `command_execution`. Best-effort (a cached
|
|
143
|
+
* skill might not be re-read); for the trigger-rate `fired` predicate over Codex.
|
|
144
|
+
*/
|
|
145
|
+
function codexSkillFired(run, name) {
|
|
146
|
+
const needle = `${name}/SKILL.md`;
|
|
147
|
+
return run.toolCalls.some((c) => str(c.name).includes(needle));
|
|
148
|
+
}
|
|
149
|
+
/**
|
|
150
|
+
* Materialize a (Claude-shaped) plugin dir's skills into `<cwd>/.codex/skills/` β
|
|
151
|
+
* where Codex actually discovers them (validated live: codex reads
|
|
152
|
+
* `<cwd>/.codex/skills/<name>/SKILL.md`). This is the Codex analog of Claude's
|
|
153
|
+
* `--plugin-dir`: `measureTriggerRate` hands the runner a `pluginDir` (the
|
|
154
|
+
* stubbed/packaged skills), and the Codex runner installs them here before the
|
|
155
|
+
* turn. Pure fs β unit-testable without a binary.
|
|
156
|
+
*/
|
|
157
|
+
function installCodexSkills(pluginDir, cwd) {
|
|
158
|
+
const skillsRoot = (0, node_path_1.join)(pluginDir, "skills");
|
|
159
|
+
if (!(0, node_fs_1.existsSync)(skillsRoot))
|
|
160
|
+
return 0;
|
|
161
|
+
let n = 0;
|
|
162
|
+
for (const name of (0, node_fs_1.readdirSync)(skillsRoot)) {
|
|
163
|
+
const src = (0, node_path_1.join)(skillsRoot, name, "SKILL.md");
|
|
164
|
+
if (!(0, node_fs_1.existsSync)(src))
|
|
165
|
+
continue;
|
|
166
|
+
const dest = (0, node_path_1.join)(cwd, ".codex", "skills", name, "SKILL.md");
|
|
167
|
+
(0, node_fs_1.mkdirSync)((0, node_path_1.dirname)(dest), { recursive: true });
|
|
168
|
+
(0, node_fs_1.writeFileSync)(dest, (0, node_fs_1.readFileSync)(src, "utf-8"));
|
|
169
|
+
n += 1;
|
|
170
|
+
}
|
|
171
|
+
return n;
|
|
172
|
+
}
|
|
173
|
+
/* v8 ignore start -- real codex subprocess; validated against the binary, not the unit gate */
|
|
174
|
+
/**
|
|
175
|
+
* The Codex eval-tier `AgentRunner`: install the run's skills into `.codex/skills`
|
|
176
|
+
* (Codex's discovery path, vs Claude's `--plugin-dir`), then drive a real
|
|
177
|
+
* `codex exec --json` turn. The seam `measureTriggerRate(spec, { evalDriver:
|
|
178
|
+
* codexEvalDriver })` dispatches through.
|
|
179
|
+
*/
|
|
180
|
+
function codexEvalAgentRunner(args) {
|
|
181
|
+
if (args.pluginDir)
|
|
182
|
+
installCodexSkills(args.pluginDir, args.cwd);
|
|
183
|
+
return Promise.resolve(codexEvalRunner({
|
|
184
|
+
task: args.task,
|
|
185
|
+
cwd: args.cwd,
|
|
186
|
+
timeoutMs: args.timeoutMs,
|
|
187
|
+
}));
|
|
188
|
+
}
|
|
189
|
+
/**
|
|
190
|
+
* The Codex eval driver β pass to `measureTriggerRate(spec, { evalDriver:
|
|
191
|
+
* codexEvalDriver })` to run a trigger-rate eval natively on `codex exec`. Pair
|
|
192
|
+
* the spec's `fired` with `codexSkillFired` (Codex has no Skill-tool event).
|
|
193
|
+
*/
|
|
194
|
+
exports.codexEvalDriver = {
|
|
195
|
+
runner: codexEvalAgentRunner,
|
|
196
|
+
parse: parseCodexEvalRun,
|
|
197
|
+
runError: codexRunError,
|
|
198
|
+
};
|
|
199
|
+
/**
|
|
200
|
+
* Spawn real `codex exec --json` for the eval tier (real model, the user's codex
|
|
201
|
+
* auth β NOT the mock). CONFIRMED flags (codex 0.139.0): `--json` for the event
|
|
202
|
+
* stream, `--skip-git-repo-check` for a bare cwd, the approvals/sandbox bypass so
|
|
203
|
+
* the turn runs unattended, `-C <cwd>` for the working dir, prompt as the trailing
|
|
204
|
+
* positional, and stdin = /dev/null (`stdio: ["ignore",β¦]`) β codex otherwise
|
|
205
|
+
* blocks on "Reading additional input from stdinβ¦". Needs ChatGPT/API auth +
|
|
206
|
+
* network egress to the model backend.
|
|
207
|
+
*/
|
|
208
|
+
function codexEvalRunner(args) {
|
|
209
|
+
const r = (0, node_child_process_1.spawnSync)("codex", [
|
|
210
|
+
"exec",
|
|
211
|
+
"--json",
|
|
212
|
+
"--skip-git-repo-check",
|
|
213
|
+
"--dangerously-bypass-approvals-and-sandbox",
|
|
214
|
+
"-C",
|
|
215
|
+
args.cwd,
|
|
216
|
+
args.task,
|
|
217
|
+
], {
|
|
218
|
+
cwd: args.cwd,
|
|
219
|
+
encoding: "utf-8",
|
|
220
|
+
timeout: args.timeoutMs,
|
|
221
|
+
stdio: ["ignore", "pipe", "pipe"],
|
|
222
|
+
maxBuffer: 64 * 1024 * 1024,
|
|
223
|
+
});
|
|
224
|
+
return { code: r.status ?? 1, stdout: r.stdout ?? "" };
|
|
225
|
+
}
|
|
226
|
+
/* v8 ignore stop */
|
|
227
|
+
//# sourceMappingURL=eval.js.map
|