@fede0089/skill-eval 1.4.1 → 3.0.0

package/README.md

@@ -2,11 +2,18 @@
 
 A CLI tool for evaluating Agent Skills locally. Tests whether your skill triggers reliably and produces the right output, using an LLM as the judge.
 
-## Why skill evals?
+## What you can test
 
-Skills are instructions that change how the agent behaves. But a single successful run isn't enough to trust one — agents are non-deterministic, and a good isolated result can be exactly that: an isolated case.
+skill-eval ships two commands, each targeting a different failure mode:
 
-Skill evals let you turn that intuition into evidence: run several comparable tasks with the skill, measure them against the same criteria, and optionally compare against a baseline or historical skill branches.
+- **Triggering** (`skill-eval trigger`) — checks whether the agent actually decides to invoke the skill in the right context. A skill that never gets triggered cannot help, no matter how good its instructions are.
+- **Functional correctness** (`skill-eval functional`) — checks whether the actions the agent takes while the skill is active match your expectations. An LLM judge grades each transcript against the expectation list you provide.
+
+## Why run skill evals
+
+- **Avoid regression** — it is very common that, while iterating to make a skill handle a new case, it stops solving cases it used to handle. Evals make every iteration measurable, so you can tell whether a change adds value without silently subtracting it elsewhere.
+- **Validate against a baseline** — sometimes an agent solves a task better using its general capabilities than using a specific skill (and even if it works today, model upgrades can shift that balance). Comparing against the no-skill baseline (`--compare-baseline`) or past skill versions (`--compare-ref`) tells you whether the skill is really pulling its weight.
+- **Statistical confidence** — LLMs are non-deterministic, so a single passing run is not evidence. Running the same expectations N times produces a pass rate (pass@k) that turns "it feels like it works" into a number you can defend.
 
 ## How it works
 
@@ -37,7 +44,7 @@ For each eval prompt, skill-eval spins up parallel agent processes with the curr
 
 ## Installation
 
-**Requirements:** Node.js, and the agent CLI you want to evaluate (e.g. `gemini`) installed and on `$PATH`.
+**Requirements:** Node.js, and the agent CLI you want to evaluate (e.g. `gemini`, `codex`, or `claude`) installed and on `$PATH`.
 
 ### Run without installing
 
@@ -87,6 +94,12 @@ skill-eval functional --workspace <path> --skill <path> [options] [agent]
 | `-v, --debug` | no | `false` | Enable verbose debug logging |
 | `[agent]` | no | `gemini-cli` | Agent backend to use |
 
+Supported runners:
+
+- `gemini-cli` (default)
+- `codex`
+- `claude-code`
+
 ### Skill directory structure
 
 ```
@@ -95,8 +108,12 @@ my-skill/
 └── evals/                          # evaluation suite (required)
     ├── my-evals.json               # one or more eval files (*.json)
     └── config/                     # runner configuration (optional but often needed)
-        └── gemini-cli/             # runner-specific config folder
-            └── settings.json       # copied to <worktree>/.gemini/ before each trial
+        ├── gemini-cli/             # copied to <worktree>/.gemini/ before each trial
+        │   └── settings.json
+        ├── codex/                  # copied to <worktree>/.codex/ before each trial
+        │   └── config.toml
+        └── claude-code/            # copied to <worktree>/.claude/ before each trial
+            └── settings.json
 ```
 
 All `.json` files in `evals/` are loaded and merged into a single suite — you can split them by feature or regression category.
@@ -134,15 +151,34 @@ All `.json` files in `evals/` are loaded and merged into a single suite — you
 
 skill-eval runs the agent headlessly — stdin is closed, there is no terminal. If the agent encounters a tool that requires interactive approval, it will either fail immediately or hang until the trial timeout kills it.
 
-The runner already uses `--approval-mode auto_edit`, which auto-approves standard file operations (create, edit, delete). But if your skill needs to run shell commands, read environment variables, make network calls, or use any other tool category — those still require explicit permission.
+Each runner configures its own non-interactive mode. For example, Gemini CLI uses `--approval-mode auto_edit`, Codex uses `codex exec --json --sandbox workspace-write -c approval_policy="never"`, and Claude Code uses `claude -p --output-format stream-json --permission-mode bypassPermissions`. If your skill needs to run shell commands, read environment variables, make network calls, or use any other tool category, refer to that runner's permission model.
 
 **Solution:** place a config file inside your skill at `evals/config/<runner>/`. Before every trial, skill-eval automatically copies that directory into the agent's config location inside the isolated worktree:
 
 ```
-evals/config/gemini-cli/  →  <worktree>/.gemini/
+evals/config/gemini-cli/   →  <worktree>/.gemini/
+evals/config/codex/        →  <worktree>/.codex/
+evals/config/claude-code/  →  <worktree>/.claude/
+```
+
+Use this to ship both settings and policies alongside your evals. Anything inside `evals/config/<runner>/` is dropped verbatim into the runner's config directory, so you can use the runner's full configuration surface — not just `settings.json`.
+
+### Gemini CLI example
+
+Gemini CLI reads both `settings.json` and any `*.toml` rule files under `policies/`. Drop them inside `evals/config/gemini-cli/` and skill-eval will copy them into `<worktree>/.gemini/` before each trial:
+
+```
+my-skill/
+└── evals/
+    └── config/
+        └── gemini-cli/
+            ├── settings.json
+            └── policies/
+                ├── allow-activate-skill.toml
+                └── allow-tools.toml
 ```
 
-Use this to ship both settings and policies alongside your evals. For Gemini CLI, for example, you can use `settings.json` to configure tool permissions and approval policies so that every tool your skill relies on runs without prompting:
+`settings.json` holds general configuration (telemetry, model, etc.):
 
 ```json
 {
@@ -150,10 +186,46 @@ Use this to ship both settings and policies alongside your evals. For Gemini CLI
 }
 ```
 
-Refer to your runner's documentation for the full list of available settings and policy keys.
+Policies whitelist specific tools so they run without prompting. For example, to always allow the `activate_skill` dispatch tool in non-interactive mode:
+
+```toml
+# evals/config/gemini-cli/policies/allow-activate-skill.toml
+[[rule]]
+toolName    = "activate_skill"
+decision    = "allow"
+priority    = 100
+interactive = false
+```
+
+Refer to your runner's documentation for the full set of settings and policy keys (Codex uses `config.toml`, Claude Code uses `settings.json`).
 
 > This config only applies inside the temporary worktree created for each trial. Your real workspace config is never touched.
 
+## Reports
+
+Each run writes to `.project-skill-evals/runs/<timestamp>/` and includes per-trial logs, the raw eval JSON, and a self-contained HTML report you can open in any browser. The report shows pass@k aggregates per eval, lets you expand each trial, and color-codes triggering vs. functional outcomes.
+
+A published sample report is available at [fede0089.github.io/skill-eval/sample-report.html](https://fede0089.github.io/skill-eval/sample-report.html). The checked-in source is [`docs/sample-report.html`](docs/sample-report.html), with the matching screenshot at [`docs/sample-report.png`](docs/sample-report.png).
+
+It was generated from the project root with:
+
+```sh
+skill-eval functional --workspace . --skill mock-skill --trials 2 --compare-baseline --debug claude-code
+```
+
+![Sample HTML report](docs/sample-report.png)
+
+To publish or refresh the GitHub Page, commit the files under `docs/`, then enable **Settings -> Pages -> Build and deployment -> Deploy from a branch** in GitHub and select the `main` branch with the `/docs` folder. GitHub will serve the report at `https://fede0089.github.io/skill-eval/sample-report.html`.
+
+### Debug logs
+
+When a trial misbehaves, pass `-v` / `--debug` to capture the full transcripts to disk. Each trial writes a `task_<id>_<variant>_trial_<n>.log` file inside the run directory with two sections appended in order:
+
+- `# SECTION: <MODE> AGENT RUN` — the initial prompt sent to the agent and its raw streamed response.
+- `# SECTION: <MODE> JUDGE RUN` — the prompt sent to the LLM judge and its verdict (only present for `functional` runs; `trigger` is graded programmatically and produces no judge section).
+
+Without `--debug` these files are not written, so reach for the flag when you need to see exactly what the agent — or the judge — saw.
+
 ## Try it out
 
 This repo includes a `mock-skill/` directory — a complete, working example of a license-generator skill with trigger and functional evals. Run it directly with:
@@ -162,10 +234,12 @@ This repo includes a `mock-skill/` directory — a complete, working example of
 npm run test:unit        # run the unit test suite
 npm run test:trigger     # trigger evaluation against mock-skill
 npm run test:functional  # functional evaluation against mock-skill
+npm run test:trigger -- codex            # run trigger evals with Codex
+npm run test:functional -- codex         # run functional evals with Codex
+npm run test:trigger -- claude-code      # run trigger evals with Claude Code
+npm run test:functional -- claude-code   # run functional evals with Claude Code
 ```
 
-Results are saved to `.project-skill-evals/runs/<timestamp>/` with logs, raw eval JSONs, and an HTML report.
-
 ## Extending
 
 ### Adding a new agent runner

package/dist/core/environment.js

@@ -37,9 +37,11 @@ export class EvalEnvironment {
      */
     createWorktree(evalId) {
         const worktreePath = path.resolve(this.workspace, '.project-skill-evals', 'worktrees', evalId);
+        const branchName = path.basename(worktreePath);
         // Ensure the path is clean before adding a worktree.
         // We try to remove it first in case a previous run crashed.
         executor.spawnSync('git', ['worktree', 'remove', '--force', worktreePath], { stdio: 'ignore', cwd: this.workspace });
+        executor.spawnSync('git', ['branch', '-D', branchName], { stdio: 'ignore', cwd: this.workspace });
         // If git worktree remove failed (e.g. path was never registered, or git
         // metadata is stale), fall back to a physical wipe and a metadata prune so
         // that 'git worktree add' does not exit 128 on a pre-existing path.

package/dist/runners/claude-code/index.js

@@ -0,0 +1 @@
+export { ClaudeCodeRunner } from './runner.js';

package/dist/runners/claude-code/runner.js

@@ -0,0 +1,279 @@
+import * as fs from 'fs';
+import * as path from 'path';
+import child_process from 'child_process';
+import { Logger } from '../../utils/logger.js';
+function isRecord(value) {
+    return typeof value === 'object' && value !== null;
+}
+function readSkillName(skillPath) {
+    const skillMd = path.join(skillPath, 'SKILL.md');
+    if (!fs.existsSync(skillMd))
+        return path.basename(skillPath);
+    const content = fs.readFileSync(skillMd, 'utf-8');
+    const frontmatter = content.match(/^---\n([\s\S]*?)\n---/);
+    if (!frontmatter)
+        return path.basename(skillPath);
+    const nameLine = frontmatter[1].split('\n').find((line) => line.trim().startsWith('name:'));
+    const name = nameLine?.replace(/^name:\s*/, '').trim().replace(/^["']|["']$/g, '');
+    return name || path.basename(skillPath);
+}
+function numberFrom(value) {
+    return typeof value === 'number' ? value : undefined;
+}
+function getAssistantContent(event) {
+    const eventType = String(event.type ?? '');
+    if (eventType !== 'assistant')
+        return [];
+    const message = event.message;
+    if (!isRecord(message))
+        return [];
+    const content = message.content;
+    if (!Array.isArray(content))
+        return [];
+    return content.filter(isRecord);
+}
+function getUserContent(event) {
+    const eventType = String(event.type ?? '');
+    if (eventType !== 'user')
+        return [];
+    const message = event.message;
+    if (!isRecord(message))
+        return [];
+    const content = message.content;
+    if (!Array.isArray(content))
+        return [];
+    return content.filter(isRecord);
+}
+/**
+ * Converts Claude Code stream-json events into the small NDJSON contract used
+ * by the rest of the evaluator. The normalizer is intentionally tolerant: the
+ * Claude Code event stream is richer than the evaluator needs and may grow
+ * over time. Only fields the grader/judge consume are mapped.
+ */
+export function normalizeClaudeJsonl(output) {
+    const normalized = [];
+    let sawResult = false;
+    for (const line of output.split('\n')) {
+        const trimmed = line.trim();
+        if (!trimmed)
+            continue;
+        let event;
+        try {
+            event = JSON.parse(trimmed);
+        }
+        catch {
+            continue;
+        }
+        for (const item of getAssistantContent(event)) {
+            const itemType = String(item.type ?? '');
+            if (itemType === 'text' && typeof item.text === 'string' && item.text.trim()) {
+                normalized.push({ type: 'message', role: 'assistant', content: item.text });
+            }
+            else if (itemType === 'tool_use') {
+                const toolName = String(item.name ?? '');
+                const toolId = String(item.id ?? `claude-tool-${normalized.length + 1}`);
+                const input = isRecord(item.input) ? item.input : {};
+                // The internal NDJSON contract expects parameters.name to carry the skill
+                // identifier when the dispatch tool is invoked. Claude Code's Skill tool
+                // uses `input.skill` for that — remap so TriggerGrader can match it.
+                const parameters = { ...input };
+                if (toolName === 'Skill' && typeof input.skill === 'string') {
+                    parameters.name = input.skill;
+                }
+                normalized.push({
+                    type: 'tool_use',
+                    tool_id: toolId,
+                    tool_name: toolName,
+                    parameters,
+                });
+            }
+        }
+        for (const item of getUserContent(event)) {
+            const itemType = String(item.type ?? '');
+            if (itemType !== 'tool_result')
+                continue;
+            const toolId = String(item.tool_use_id ?? '');
+            if (!toolId)
+                continue;
+            const isError = item.is_error === true;
+            // Claude Code attaches a tool_use_result sibling on the parent event with
+            // a `success` boolean for built-in tools (e.g. Skill). Prefer that when present.
+            const parent = event.tool_use_result;
+            const parentSuccess = isRecord(parent) ? parent.success : undefined;
+            const status = isError === true || parentSuccess === false ? 'error' : 'success';
+            normalized.push({ type: 'tool_result', tool_id: toolId, status });
+        }
+        const eventType = String(event.type ?? '');
+        if (eventType === 'result') {
+            sawResult = true;
+            const isError = event.is_error === true;
+            const usage = isRecord(event.usage) ? event.usage : undefined;
+            const inputTokens = usage ? numberFrom(usage.input_tokens) ?? 0 : 0;
+            const outputTokens = usage ? numberFrom(usage.output_tokens) ?? 0 : 0;
+            const cachedTokens = usage
+                ? numberFrom(usage.cache_read_input_tokens)
+                    ?? numberFrom(usage.cached_input_tokens)
+                    ?? 0
+                : 0;
+            const stats = usage
+                ? {
+                    total_tokens: inputTokens + outputTokens,
+                    input_tokens: inputTokens,
+                    output_tokens: outputTokens,
+                    cached: cachedTokens,
+                }
+                : undefined;
+            if (isError) {
+                const message = typeof event.result === 'string' && event.result.trim()
+                    ? event.result
+                    : `Claude Code run failed (subtype: ${String(event.subtype ?? 'unknown')})`;
+                normalized.push({ type: 'result', status: 'error', error: { message }, ...(stats ? { stats } : {}) });
+            }
+            else {
+                const response = typeof event.result === 'string' ? event.result : undefined;
+                normalized.push({
+                    type: 'result',
+                    status: 'success',
+                    ...(response ? { response } : {}),
+                    ...(stats ? { stats } : {}),
+                });
+            }
+        }
+    }
+    if (!sawResult) {
+        normalized.push({ type: 'result', status: 'error', error: { message: 'Claude Code produced no result event' } });
+    }
+    return normalized.map((event) => JSON.stringify(event)).join('\n');
+}
+export class ClaudeCodeRunner {
+    skillDispatchToolName = 'Skill';
+    linkedSkillsByWorktree = new Map();
+    async runPrompt(prompt, cwd, onLog, logPath, extraArgs = [], timeoutMs) {
+        return new Promise((resolve) => {
+            let stdout = '';
+            let stderr = '';
+            let resolved = false;
+            let timeout;
+            const args = [
+                '-p', prompt,
+                '--output-format', 'stream-json',
+                '--verbose',
+                '--permission-mode', 'bypassPermissions',
+                '--no-session-persistence',
+                ...extraArgs,
+            ];
+            const spawnOptions = {
+                cwd,
+                env: { ...process.env, FORCE_COLOR: '0' },
+                stdio: ['ignore', 'pipe', 'pipe'],
+                detached: true,
+            };
+            function appendLog(chunk) {
+                if (!logPath)
+                    return;
+                try {
+                    fs.appendFileSync(logPath, chunk);
+                }
+                catch (err) {
+                    Logger.warn(`Failed to write Claude Code debug log at ${logPath}. Continuing. Reason: ${err}`);
+                }
+            }
+            function killProcessGroup() {
+                if (!child.pid)
+                    return;
+                try {
+                    process.kill(-child.pid, 'SIGKILL');
+                }
+                catch {
+                    // Process might have already exited.
+                }
+            }
+            const child = child_process.spawn('claude', args, spawnOptions);
+            appendLog(`--- Claude Code Execution Start: ${new Date().toISOString()} ---\n`);
+            appendLog(`Command: claude ${args.join(' ')}\n\n`);
+            if (timeoutMs && timeoutMs > 0) {
+                timeout = setTimeout(() => {
+                    if (resolved)
+                        return;
+                    resolved = true;
+                    killProcessGroup();
+                    const timeoutSec = timeoutMs / 1000;
+                    Logger.error(`\nClaude Code process timed out after ${timeoutSec} seconds.`);
+                    appendLog(`\n\n--- Claude Code process timed out ---\n${stderr}\n`);
+                    resolve({ error: `Process timeout exceeded (${timeoutSec} seconds)`, raw_output: stderr });
+                }, timeoutMs);
+            }
+            child.stdout?.on('data', (data) => {
+                const chunk = data.toString();
+                stdout += chunk;
+                appendLog(chunk);
+            });
+            child.stderr?.on('data', (data) => {
+                const chunk = data.toString();
+                stderr += chunk;
+                if (onLog) {
+                    const lines = chunk.split('\n').filter((line) => line.trim() !== '');
+                    if (lines.length > 0)
+                        onLog(lines[lines.length - 1]);
+                }
+            });
+            child.on('error', (err) => {
+                if (resolved)
+                    return;
+                resolved = true;
+                if (timeout)
+                    clearTimeout(timeout);
+                Logger.error(`Failed to start Claude Code CLI. Error: ${err.message}`);
+                appendLog(`\n\n--- Error starting Claude Code CLI ---\n${err.message}\n`);
+                resolve(null);
+            });
+            child.on('close', (code) => {
+                appendLog(`\n\n--- Claude Code exited with status ${code} ---\n`);
+                if (code !== 0 && stderr) {
+                    appendLog(`--- Stderr ---\n${stderr}\n--- End Stderr ---\n`);
+                }
+                if (resolved)
+                    return;
+                resolved = true;
+                if (timeout)
+                    clearTimeout(timeout);
+                if (!stdout.trim()) {
+                    resolve({ error: 'Empty output from Claude Code CLI', raw_output: stderr });
+                    return;
+                }
+                const normalized = normalizeClaudeJsonl(stdout);
+                if (code !== 0) {
+                    resolve({
+                        error: `Claude Code CLI exited with status ${code}`,
+                        response: normalized,
+                        raw_output: `${normalized}\n--- CLAUDE CODE STDOUT ---\n${stdout}\n--- STDERR ---\n${stderr}`,
+                    });
+                    return;
+                }
+                resolve({
+                    response: normalized,
+                    raw_output: `${normalized}\n--- CLAUDE CODE STDOUT ---\n${stdout}\n--- STDERR ---\n${stderr}`,
+                });
+            });
+        });
+    }
+    applyRunnerConfig(evalConfigBaseDir, worktreePath) {
+        const src = path.join(evalConfigBaseDir, 'claude-code');
+        if (!fs.existsSync(src))
+            return;
+        const dst = path.join(worktreePath, '.claude');
+        fs.mkdirSync(dst, { recursive: true });
+        fs.cpSync(src, dst, { recursive: true, force: true });
+    }
+    async linkSkill(absoluteSkillPath, worktreePath) {
+        const skillName = readSkillName(absoluteSkillPath);
+        const localSkillsDir = path.join(worktreePath, '.claude', 'skills');
+        const symlinkPath = path.join(localSkillsDir, skillName);
+        fs.mkdirSync(localSkillsDir, { recursive: true });
+        if (fs.existsSync(symlinkPath)) {
+            fs.rmSync(symlinkPath, { recursive: true, force: true });
+        }
+        fs.symlinkSync(absoluteSkillPath, symlinkPath, 'dir');
+        this.linkedSkillsByWorktree.set(path.resolve(worktreePath), { name: skillName });
+    }
+}

package/dist/runners/codex/index.js

@@ -0,0 +1 @@
+export { CodexRunner } from './runner.js';

package/dist/runners/codex/runner.js

@@ -0,0 +1,330 @@
+import * as fs from 'fs';
+import * as path from 'path';
+import child_process from 'child_process';
+import { Logger } from '../../utils/logger.js';
+function isRecord(value) {
+    return typeof value === 'object' && value !== null;
+}
+function readSkillName(skillPath) {
+    const skillMd = path.join(skillPath, 'SKILL.md');
+    if (!fs.existsSync(skillMd))
+        return path.basename(skillPath);
+    const content = fs.readFileSync(skillMd, 'utf-8');
+    const frontmatter = content.match(/^---\n([\s\S]*?)\n---/);
+    if (!frontmatter)
+        return path.basename(skillPath);
+    const nameLine = frontmatter[1].split('\n').find((line) => line.trim().startsWith('name:'));
+    const name = nameLine?.replace(/^name:\s*/, '').trim().replace(/^["']|["']$/g, '');
+    return name || path.basename(skillPath);
+}
+function tomlString(value) {
+    return JSON.stringify(value);
+}
+function extractText(value) {
+    if (typeof value === 'string')
+        return [value];
+    if (Array.isArray(value))
+        return value.flatMap(extractText);
+    if (!isRecord(value))
+        return [];
+    const direct = ['text', 'message', 'content', 'output_text', 'final_message']
+        .flatMap((key) => extractText(value[key]));
+    if (direct.length > 0)
+        return direct;
+    return [];
+}
+function getEventItem(event) {
+    const item = event.item ?? event.message ?? event.event;
+    return isRecord(item) ? item : undefined;
+}
+function getAssistantText(event) {
+    const eventType = String(event.type ?? '');
+    if (eventType === 'agent_message' || eventType === 'assistant_message') {
+        return extractText(event);
+    }
+    const item = getEventItem(event);
+    if (!item)
+        return [];
+    const itemType = String(item.type ?? '');
+    if (itemType === 'agent_message' || itemType === 'assistant_message' || itemType === 'message') {
+        return extractText(item);
+    }
+    return [];
+}
+function eventSignalsSkill(event, skillName) {
+    const serialized = JSON.stringify(event).toLowerCase();
+    const normalizedSkill = skillName.toLowerCase();
+    const eventType = String(event.type ?? '').toLowerCase();
+    const item = getEventItem(event);
+    const itemType = String(item?.type ?? '').toLowerCase();
+    return (serialized.includes(normalizedSkill) &&
+        (serialized.includes('skill') ||
+            eventType.includes('skill') ||
+            itemType.includes('skill') ||
+            serialized.includes('"activate_skill"') ||
+            serialized.includes('"use_skill"')));
+}
+function eventIsFailure(event) {
+    const status = String(event.status ?? '').toLowerCase();
+    const item = getEventItem(event);
+    const itemStatus = String(item?.status ?? '').toLowerCase();
+    return ['error', 'failed', 'failure'].includes(status) || ['error', 'failed', 'failure'].includes(itemStatus);
+}
+function getErrorMessage(event) {
+    const eventType = String(event.type ?? '').toLowerCase();
+    if (!eventType.includes('error') && !eventIsFailure(event))
+        return null;
+    const message = extractText(event).join('\n').trim();
+    if (message)
+        return message;
+    const error = event.error;
+    if (typeof error === 'string')
+        return error;
+    if (isRecord(error)) {
+        const nested = extractText(error).join('\n').trim();
+        if (nested)
+            return nested;
+    }
+    return 'Codex run failed';
+}
+function getUsage(event) {
+    const usage = event.usage ?? event.token_usage;
+    if (isRecord(usage))
+        return usage;
+    const item = getEventItem(event);
+    const itemUsage = item?.usage ?? item?.token_usage;
+    return isRecord(itemUsage) ? itemUsage : undefined;
+}
+function numberFrom(value) {
+    return typeof value === 'number' ? value : undefined;
+}
+/**
+ * Converts Codex exec JSONL events into the small NDJSON contract used by the
+ * rest of the evaluator. The normalizer is intentionally tolerant because the
+ * Codex JSON event stream is richer than the evaluator needs and may grow over
+ * time.
+ */
+export function normalizeCodexJsonl(output, skillName) {
+    const normalized = [];
+    let sawResult = false;
+    let sawSkillActivation = false;
+    let errorMessage = null;
+    let usage;
+    for (const line of output.split('\n')) {
+        const trimmed = line.trim();
+        if (!trimmed)
+            continue;
+        let event;
+        try {
+            event = JSON.parse(trimmed);
+        }
+        catch {
+            continue;
+        }
+        for (const text of getAssistantText(event)) {
+            if (text.trim()) {
+                normalized.push({ type: 'message', role: 'assistant', content: text });
+            }
+        }
+        if (skillName && !sawSkillActivation && eventSignalsSkill(event, skillName)) {
+            sawSkillActivation = true;
+            normalized.push({
+                type: 'tool_use',
+                tool_id: `codex-skill-${normalized.length + 1}`,
+                tool_name: 'activate_skill',
+                parameters: { name: skillName },
+            });
+            normalized.push({
+                type: 'tool_result',
+                tool_id: `codex-skill-${normalized.length}`,
+                status: eventIsFailure(event) ? 'error' : 'success',
+            });
+        }
+        const eventUsage = getUsage(event);
+        if (eventUsage)
+            usage = eventUsage;
+        const maybeError = getErrorMessage(event);
+        if (maybeError)
+            errorMessage = maybeError;
+        const eventType = String(event.type ?? '');
+        if (eventType === 'result') {
+            sawResult = true;
+            normalized.push(event);
+        }
+    }
+    if (!sawResult) {
+        if (errorMessage) {
+            normalized.push({ type: 'result', status: 'error', error: { message: errorMessage } });
+        }
+        else {
+            const inputTokens = usage ? numberFrom(usage.input_tokens) ?? numberFrom(usage.inputTokens) ?? 0 : 0;
+            const outputTokens = usage ? numberFrom(usage.output_tokens) ?? numberFrom(usage.outputTokens) ?? 0 : 0;
+            const stats = usage ? {
+                total_tokens: numberFrom(usage.total_tokens) ?? numberFrom(usage.totalTokens) ?? inputTokens + outputTokens,
+                input_tokens: inputTokens,
+                output_tokens: outputTokens,
+                cached: numberFrom(usage.cached) ?? numberFrom(usage.cached_tokens) ?? numberFrom(usage.cached_input_tokens) ?? numberFrom(usage.cachedTokens) ?? 0,
+            } : undefined;
+            normalized.push({ type: 'result', status: 'success', ...(stats ? { stats } : {}) });
+        }
+    }
+    return normalized.map((event) => JSON.stringify(event)).join('\n');
+}
+export class CodexRunner {
+    skillDispatchToolName = 'activate_skill';
+    linkedSkillsByWorktree = new Map();
+    async runPrompt(prompt, cwd, onLog, logPath, extraArgs = [], timeoutMs) {
+        return new Promise((resolve) => {
+            let stdout = '';
+            let stderr = '';
+            let resolved = false;
+            let timeout;
+            const linkedSkill = cwd ? this.linkedSkillsByWorktree.get(path.resolve(cwd)) : undefined;
+            const skillConfigArgs = linkedSkill
+                ? ['-c', `skills.config=[{path=${tomlString(linkedSkill.path)},enabled=true}]`]
+                : [];
+            const instructionArgs = linkedSkill
+                ? ['-c', `model_instructions_file=${tomlString(linkedSkill.instructionsPath)}`]
+                : [];
+            const args = [
+                'exec',
+                '--json',
+                '--cd', cwd ?? process.cwd(),
+                '--sandbox', 'workspace-write',
+                '-c', 'approval_policy="never"',
+                '--skip-git-repo-check',
+                '--ephemeral',
+                '--color', 'never',
+                ...skillConfigArgs,
+                ...instructionArgs,
+                ...extraArgs,
+                prompt,
+            ];
+            const spawnOptions = {
+                cwd,
+                env: { ...process.env, FORCE_COLOR: '0' },
+                stdio: ['ignore', 'pipe', 'pipe'],
+                detached: true,
+            };
+            function appendLog(chunk) {
+                if (!logPath)
+                    return;
+                try {
+                    fs.appendFileSync(logPath, chunk);
+                }
+                catch (err) {
+                    Logger.warn(`Failed to write Codex debug log at ${logPath}. Continuing. Reason: ${err}`);
+                }
+            }
+            function killProcessGroup() {
+                if (!child.pid)
+                    return;
+                try {
+                    process.kill(-child.pid, 'SIGKILL');
+                }
+                catch {
+                    // Process might have already exited.
+                }
+            }
+            const child = child_process.spawn('codex', args, spawnOptions);
+            appendLog(`--- Codex Execution Start: ${new Date().toISOString()} ---\n`);
+            appendLog(`Command: codex ${args.join(' ')}\n\n`);
+            if (timeoutMs && timeoutMs > 0) {
+                timeout = setTimeout(() => {
+                    if (resolved)
+                        return;
+                    resolved = true;
+                    killProcessGroup();
+                    const timeoutSec = timeoutMs / 1000;
+                    Logger.error(`\nCodex process timed out after ${timeoutSec} seconds.`);
+                    appendLog(`\n\n--- Codex process timed out ---\n${stderr}\n`);
+                    resolve({ error: `Process timeout exceeded (${timeoutSec} seconds)`, raw_output: stderr });
+                }, timeoutMs);
+            }
+            child.stdout?.on('data', (data) => {
+                const chunk = data.toString();
+                stdout += chunk;
+                appendLog(chunk);
+            });
+            child.stderr?.on('data', (data) => {
+                const chunk = data.toString();
+                stderr += chunk;
+                if (onLog) {
+                    const lines = chunk.split('\n').filter((line) => line.trim() !== '');
+                    if (lines.length > 0)
+                        onLog(lines[lines.length - 1]);
+                }
+            });
+            child.on('error', (err) => {
+                if (resolved)
+                    return;
+                resolved = true;
+                if (timeout)
+                    clearTimeout(timeout);
+                Logger.error(`Failed to start Codex CLI. Error: ${err.message}`);
+                appendLog(`\n\n--- Error starting Codex CLI ---\n${err.message}\n`);
+                resolve(null);
+            });
+            child.on('close', (code) => {
+                appendLog(`\n\n--- Codex exited with status ${code} ---\n`);
+                if (code !== 0 && stderr) {
+                    appendLog(`--- Stderr ---\n${stderr}\n--- End Stderr ---\n`);
+                }
+                if (resolved)
+                    return;
+                resolved = true;
+                if (timeout)
+                    clearTimeout(timeout);
+                if (!stdout.trim()) {
+                    resolve({ error: 'Empty output from Codex CLI', raw_output: stderr });
+                    return;
+                }
+                const normalized = normalizeCodexJsonl(stdout, linkedSkill?.name);
+                if (code !== 0) {
+                    resolve({
+                        error: `Codex CLI exited with status ${code}`,
+                        response: normalized,
+                        raw_output: `${normalized}\n--- CODEX STDOUT ---\n${stdout}\n--- STDERR ---\n${stderr}`,
+                    });
+                    return;
+                }
+                resolve({
+                    response: normalized,
+                    raw_output: `${normalized}\n--- CODEX STDOUT ---\n${stdout}\n--- STDERR ---\n${stderr}`,
+                });
+            });
+        });
+    }
+    applyRunnerConfig(evalConfigBaseDir, worktreePath) {
+        const src = path.join(evalConfigBaseDir, 'codex');
+        if (!fs.existsSync(src))
+            return;
+        const dst = path.join(worktreePath, '.codex');
+        fs.mkdirSync(dst, { recursive: true });
+        fs.cpSync(src, dst, { recursive: true, force: true });
+    }
+    async linkSkill(absoluteSkillPath, worktreePath) {
+        const skillName = readSkillName(absoluteSkillPath);
+        const localSkillsDir = path.join(worktreePath, '.codex', 'skills');
+        const symlinkPath = path.join(localSkillsDir, skillName);
+        fs.mkdirSync(localSkillsDir, { recursive: true });
+        if (fs.existsSync(symlinkPath)) {
+            fs.rmSync(symlinkPath, { recursive: true, force: true });
+        }
+        fs.symlinkSync(absoluteSkillPath, symlinkPath, 'dir');
+        const instructionsPath = path.join(worktreePath, '.codex', 'skill-eval-instructions.md');
+        fs.writeFileSync(instructionsPath, [
+            'You are running inside an automated Agent Skill evaluation.',
+            'Complete the user task end-to-end without asking for approval, waiting for confirmation, or stopping after a plan.',
+            'Use the configured skill when it applies. If you use a skill, read and follow its SKILL.md instructions.',
+            'Make any required edits directly in the current working directory.',
+            'Keep the final response concise.',
+            '',
+        ].join('\n'));
+        this.linkedSkillsByWorktree.set(path.resolve(worktreePath), {
+            name: skillName,
+            path: symlinkPath,
+            instructionsPath,
+        });
+    }
+}

package/dist/runners/index.js

@@ -1,2 +1,4 @@
 export { GeminiCliRunner } from './gemini-cli/index.js';
+export { CodexRunner } from './codex/index.js';
+export { ClaudeCodeRunner } from './claude-code/index.js';
 export { RUNNER_REGISTRY, RunnerFactory, DEFAULT_AGENT } from './registry.js';

package/dist/runners/registry.js

@@ -1,6 +1,10 @@
 import { GeminiCliRunner } from './gemini-cli/index.js';
+import { CodexRunner } from './codex/index.js';
+import { ClaudeCodeRunner } from './claude-code/index.js';
 export const RUNNER_REGISTRY = {
     'gemini-cli': { Runner: GeminiCliRunner, binary: 'gemini' },
+    codex: { Runner: CodexRunner, binary: 'codex' },
+    'claude-code': { Runner: ClaudeCodeRunner, binary: 'claude' },
 };
 /** Default agent name used when none is specified on the CLI. */
 export const DEFAULT_AGENT = Object.keys(RUNNER_REGISTRY)[0];

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@fede0089/skill-eval",
-  "version": "1.4.1",
+  "version": "3.0.0",
   "description": "CLI to evaluate agent skills triggering and functionality",
   "main": "dist/index.js",
   "bin": {
@@ -18,8 +18,8 @@
     "prepare": "npm run build",
     "build": "tsc",
     "start": "node dist/index.js",
-    "test:trigger": "npm run build && node ./dist/index.js trigger --workspace . --skill ./mock-skill --debug",
-    "test:functional": "npm run build && node ./dist/index.js functional --workspace . --skill ./mock-skill --debug",
+    "test:trigger": "npm run build && node ./scripts/run-mock-eval.mjs trigger",
+    "test:functional": "npm run build && node ./scripts/run-mock-eval.mjs functional",
     "test:unit": "tsx --test \"tests/**/*.test.ts\""
   },
   "type": "module",