@fede0089/skill-eval 1.4.0 → 2.0.0

package/README.md

@@ -20,7 +20,7 @@ For each eval prompt, skill-eval spins up parallel agent processes with the curr
                └───────┬───────┘
                        │
            ┌───────────┴───────────┐
-      ─ with skill ─          ─ baseline ─
+      ─ with skill ─        ─ baseline (opt) ─
       ┌──────┴──────┐         ┌─────┴──────┐
     agent 1      agent 2   agent 3      agent 4
       │              │         │             │
@@ -32,10 +32,12 @@ For each eval prompt, skill-eval spins up parallel agent processes with the curr
 ```
 
 > The `trigger` command only runs with-skill trials and checks whether the skill dispatch tool was actually invoked — no judge or baseline needed.
+>
+> The baseline branch is opt-in: enable it with `--compare-baseline` (no-skill control) or `--compare-ref <ref>` (historical skill versions).
 
 ## Installation
 
-**Requirements:** Node.js, and the agent CLI you want to evaluate (e.g. `gemini`) installed and on `$PATH`.
+**Requirements:** Node.js, and the agent CLI you want to evaluate (e.g. `gemini` or `codex`) installed and on `$PATH`.
 
 ### Run without installing
 
@@ -85,6 +87,11 @@ skill-eval functional --workspace <path> --skill <path> [options] [agent]
 | `-v, --debug` | no | `false` | Enable verbose debug logging |
 | `[agent]` | no | `gemini-cli` | Agent backend to use |
 
+Supported runners:
+
+- `gemini-cli` (default)
+- `codex`
+
 ### Skill directory structure
 
 ```
@@ -93,8 +100,10 @@ my-skill/
 └── evals/                          # evaluation suite (required)
     ├── my-evals.json               # one or more eval files (*.json)
     └── config/                     # runner configuration (optional but often needed)
-        └── gemini-cli/             # runner-specific config folder
-            └── settings.json       # copied to <worktree>/.gemini/ before each trial
+        ├── gemini-cli/             # copied to <worktree>/.gemini/ before each trial
+        │   └── settings.json
+        └── codex/                  # copied to <worktree>/.codex/ before each trial
+            └── config.toml
 ```
 
 All `.json` files in `evals/` are loaded and merged into a single suite — you can split them by feature or regression category.
@@ -132,12 +141,13 @@ All `.json` files in `evals/` are loaded and merged into a single suite — you
 
 skill-eval runs the agent headlessly — stdin is closed, there is no terminal. If the agent encounters a tool that requires interactive approval, it will either fail immediately or hang until the trial timeout kills it.
 
-The runner already uses `--approval-mode auto_edit`, which auto-approves standard file operations (create, edit, delete). But if your skill needs to run shell commands, read environment variables, make network calls, or use any other tool category — those still require explicit permission.
+Each runner configures its own non-interactive mode. For example, Gemini CLI uses `--approval-mode auto_edit`, while Codex uses `codex exec --json --sandbox workspace-write -c approval_policy="never"`. If your skill needs to run shell commands, read environment variables, make network calls, or use any other tool category, refer to that runner's permission model.
 
 **Solution:** place a config file inside your skill at `evals/config/<runner>/`. Before every trial, skill-eval automatically copies that directory into the agent's config location inside the isolated worktree:
 
 ```
 evals/config/gemini-cli/  →  <worktree>/.gemini/
+evals/config/codex/       →  <worktree>/.codex/
 ```
 
 Use this to ship both settings and policies alongside your evals. For Gemini CLI, for example, you can use `settings.json` to configure tool permissions and approval policies so that every tool your skill relies on runs without prompting:
@@ -157,8 +167,11 @@ Refer to your runner's documentation for the full list of available settings and
 This repo includes a `mock-skill/` directory — a complete, working example of a license-generator skill with trigger and functional evals. Run it directly with:
 
 ```sh
+npm run test:unit        # run the unit test suite
 npm run test:trigger     # trigger evaluation against mock-skill
 npm run test:functional  # functional evaluation against mock-skill
+npm run test:trigger -- codex     # run trigger evals with Codex
+npm run test:functional -- codex  # run functional evals with Codex
 ```
 
 Results are saved to `.project-skill-evals/runs/<timestamp>/` with logs, raw eval JSONs, and an HTML report.
@@ -181,17 +194,4 @@ The factory, preflight check, and CLI all pick it up automatically.
 ### Adding a new report format
 
 1. Create `src/reporters/<format>-reporter.ts` implementing `Reporter`.
-2. Export it and add a case in `createReporter()` in `src/reporters/index.ts`.
-3. Add the format string to `ReportFormat` in `src/types/index.ts`.
-er, binary: '<cli-binary-name>' },
-   ```
-
-The factory, preflight check, and CLI all pick it up automatically.
-
-> Implement `applyRunnerConfig(evalConfigBaseDir, worktreePath)` to copy `evalConfigBaseDir/<your-agent>/` into the appropriate config directory in the worktree (e.g. `.claude/` for a Claude runner). No-op silently if the directory doesn't exist.
-
-### Adding a new report format
-
-1. Create `src/reporters/<format>-reporter.ts` implementing `Reporter`.
-2. Export it and add a case in `createReporter()` in `src/reporters/index.ts`.
-3. Add the format string to `ReportFormat` in `src/types/index.ts`.
+2. Add a case for it in `createReporter()` in `src/reporters/index.ts`.

package/dist/core/environment.js

@@ -37,9 +37,11 @@ export class EvalEnvironment {
      */
     createWorktree(evalId) {
         const worktreePath = path.resolve(this.workspace, '.project-skill-evals', 'worktrees', evalId);
+        const branchName = path.basename(worktreePath);
         // Ensure the path is clean before adding a worktree.
         // We try to remove it first in case a previous run crashed.
         executor.spawnSync('git', ['worktree', 'remove', '--force', worktreePath], { stdio: 'ignore', cwd: this.workspace });
+        executor.spawnSync('git', ['branch', '-D', branchName], { stdio: 'ignore', cwd: this.workspace });
         // If git worktree remove failed (e.g. path was never registered, or git
         // metadata is stale), fall back to a physical wipe and a metadata prune so
         // that 'git worktree add' does not exit 128 on a pre-existing path.

package/dist/index.js

@@ -8,7 +8,9 @@ import { HtmlReporter } from './reporters/index.js';
 import { DEFAULT_AGENT } from './runners/registry.js';
 import * as path from 'path';
 import * as fs from 'fs';
+import { createRequire } from 'module';
 import { fileURLToPath } from 'url';
+const pkg = createRequire(import.meta.url)('../package.json');
 export const program = new Command();
 const errorHandler = (err) => {
     if (err instanceof AppError) {
@@ -26,7 +28,7 @@ const errorHandler = (err) => {
 program
     .name('skill-eval')
     .description('CLI to evaluate agent skills triggering and functionality')
-    .version('1.0.0')
+    .version(pkg.version)
     .option('-v, --debug', 'Enable debug logging', false);
 program.on('option:debug', () => {
     process.env.DEBUG = 'true';
@@ -36,16 +38,16 @@ program
     .description('Evaluate triggering of an agent skill')
     .requiredOption('--workspace <path>', 'Path to the workspace/repo to evaluate against')
     .requiredOption('--skill <path>', 'Path to the skill directory')
-    .option('--agents <number>', 'Number of parallel agents')
-    .option('--trials <number>', 'Number of trials per task for pass@k calculation')
+    .option('--agents <number>', 'Number of parallel agents', '4')
+    .option('--trials <number>', 'Number of trials per task for pass@k calculation', '3')
     .option('--timeout <seconds>', 'Agent timeout in seconds')
     .option('--eval-id <id>', 'Run only the eval with this ID (numeric)')
     .option('--compare-ref [refs...]', 'Compare against historical git references')
     .action((agent, options) => {
     const workspace = path.resolve(options.workspace);
     const selectedAgent = agent || DEFAULT_AGENT;
-    const maxAgents = parseInt(options.agents, 10) || 4;
-    const numTrials = options.trials !== undefined ? (parseInt(options.trials, 10) || 3) : 3;
+    const maxAgents = parseInt(options.agents, 10);
+    const numTrials = parseInt(options.trials, 10);
     const timeoutMs = options.timeout ? parseInt(options.timeout, 10) * 1000 : undefined;
     const evalId = options.evalId !== undefined ? parseInt(options.evalId, 10) : undefined;
     const compareRefs = options.compareRef || [];
@@ -53,11 +55,11 @@ program
 });
 program
     .command('functional [agent]')
-    .description('Evaluate functional correctness of an agent skill based on assertions')
+    .description('Evaluate functional correctness of an agent skill against expectations')
     .requiredOption('--workspace <path>', 'Path to the workspace/repo to evaluate against')
     .requiredOption('--skill <path>', 'Path to the skill directory')
-    .option('--agents <number>', 'Number of parallel agents')
-    .option('--trials <number>', 'Number of trials per task for pass@k calculation')
+    .option('--agents <number>', 'Number of parallel agents', '4')
+    .option('--trials <number>', 'Number of trials per task for pass@k calculation', '3')
     .option('--timeout <seconds>', 'Agent timeout in seconds')
     .option('--eval-id <id>', 'Run only the eval with this ID (numeric)')
     .option('--compare-ref [refs...]', 'Compare against historical git references')
@@ -65,8 +67,8 @@ program
     .action((agent, options) => {
     const workspace = path.resolve(options.workspace);
     const selectedAgent = agent || DEFAULT_AGENT;
-    const maxAgents = parseInt(options.agents, 10) || 4;
-    const numTrials = options.trials !== undefined ? (parseInt(options.trials, 10) || 3) : 3;
+    const maxAgents = parseInt(options.agents, 10);
+    const numTrials = parseInt(options.trials, 10);
     const timeoutMs = options.timeout ? parseInt(options.timeout, 10) * 1000 : undefined;
     const evalId = options.evalId !== undefined ? parseInt(options.evalId, 10) : undefined;
     const compareRefs = options.compareRef || [];

package/dist/runners/codex/index.js

@@ -0,0 +1 @@
+export { CodexRunner } from './runner.js';

package/dist/runners/codex/runner.js

@@ -0,0 +1,330 @@
+import * as fs from 'fs';
+import * as path from 'path';
+import child_process from 'child_process';
+import { Logger } from '../../utils/logger.js';
+function isRecord(value) {
+    return typeof value === 'object' && value !== null;
+}
+function readSkillName(skillPath) {
+    const skillMd = path.join(skillPath, 'SKILL.md');
+    if (!fs.existsSync(skillMd))
+        return path.basename(skillPath);
+    const content = fs.readFileSync(skillMd, 'utf-8');
+    const frontmatter = content.match(/^---\n([\s\S]*?)\n---/);
+    if (!frontmatter)
+        return path.basename(skillPath);
+    const nameLine = frontmatter[1].split('\n').find((line) => line.trim().startsWith('name:'));
+    const name = nameLine?.replace(/^name:\s*/, '').trim().replace(/^["']|["']$/g, '');
+    return name || path.basename(skillPath);
+}
+function tomlString(value) {
+    return JSON.stringify(value);
+}
+function extractText(value) {
+    if (typeof value === 'string')
+        return [value];
+    if (Array.isArray(value))
+        return value.flatMap(extractText);
+    if (!isRecord(value))
+        return [];
+    const direct = ['text', 'message', 'content', 'output_text', 'final_message']
+        .flatMap((key) => extractText(value[key]));
+    if (direct.length > 0)
+        return direct;
+    return [];
+}
+function getEventItem(event) {
+    const item = event.item ?? event.message ?? event.event;
+    return isRecord(item) ? item : undefined;
+}
+function getAssistantText(event) {
+    const eventType = String(event.type ?? '');
+    if (eventType === 'agent_message' || eventType === 'assistant_message') {
+        return extractText(event);
+    }
+    const item = getEventItem(event);
+    if (!item)
+        return [];
+    const itemType = String(item.type ?? '');
+    if (itemType === 'agent_message' || itemType === 'assistant_message' || itemType === 'message') {
+        return extractText(item);
+    }
+    return [];
+}
+function eventSignalsSkill(event, skillName) {
+    const serialized = JSON.stringify(event).toLowerCase();
+    const normalizedSkill = skillName.toLowerCase();
+    const eventType = String(event.type ?? '').toLowerCase();
+    const item = getEventItem(event);
+    const itemType = String(item?.type ?? '').toLowerCase();
+    return (serialized.includes(normalizedSkill) &&
+        (serialized.includes('skill') ||
+            eventType.includes('skill') ||
+            itemType.includes('skill') ||
+            serialized.includes('"activate_skill"') ||
+            serialized.includes('"use_skill"')));
+}
+function eventIsFailure(event) {
+    const status = String(event.status ?? '').toLowerCase();
+    const item = getEventItem(event);
+    const itemStatus = String(item?.status ?? '').toLowerCase();
+    return ['error', 'failed', 'failure'].includes(status) || ['error', 'failed', 'failure'].includes(itemStatus);
+}
+function getErrorMessage(event) {
+    const eventType = String(event.type ?? '').toLowerCase();
+    if (!eventType.includes('error') && !eventIsFailure(event))
+        return null;
+    const message = extractText(event).join('\n').trim();
+    if (message)
+        return message;
+    const error = event.error;
+    if (typeof error === 'string')
+        return error;
+    if (isRecord(error)) {
+        const nested = extractText(error).join('\n').trim();
+        if (nested)
+            return nested;
+    }
+    return 'Codex run failed';
+}
+function getUsage(event) {
+    const usage = event.usage ?? event.token_usage;
+    if (isRecord(usage))
+        return usage;
+    const item = getEventItem(event);
+    const itemUsage = item?.usage ?? item?.token_usage;
+    return isRecord(itemUsage) ? itemUsage : undefined;
+}
+function numberFrom(value) {
+    return typeof value === 'number' ? value : undefined;
+}
+/**
+ * Converts Codex exec JSONL events into the small NDJSON contract used by the
+ * rest of the evaluator. The normalizer is intentionally tolerant because the
+ * Codex JSON event stream is richer than the evaluator needs and may grow over
+ * time.
+ */
+export function normalizeCodexJsonl(output, skillName) {
+    const normalized = [];
+    let sawResult = false;
+    let sawSkillActivation = false;
+    let errorMessage = null;
+    let usage;
+    for (const line of output.split('\n')) {
+        const trimmed = line.trim();
+        if (!trimmed)
+            continue;
+        let event;
+        try {
+            event = JSON.parse(trimmed);
+        }
+        catch {
+            continue;
+        }
+        for (const text of getAssistantText(event)) {
+            if (text.trim()) {
+                normalized.push({ type: 'message', role: 'assistant', content: text });
+            }
+        }
+        if (skillName && !sawSkillActivation && eventSignalsSkill(event, skillName)) {
+            sawSkillActivation = true;
+            normalized.push({
+                type: 'tool_use',
+                tool_id: `codex-skill-${normalized.length + 1}`,
+                tool_name: 'activate_skill',
+                parameters: { name: skillName },
+            });
+            normalized.push({
+                type: 'tool_result',
+                tool_id: `codex-skill-${normalized.length}`,
+                status: eventIsFailure(event) ? 'error' : 'success',
+            });
+        }
+        const eventUsage = getUsage(event);
+        if (eventUsage)
+            usage = eventUsage;
+        const maybeError = getErrorMessage(event);
+        if (maybeError)
+            errorMessage = maybeError;
+        const eventType = String(event.type ?? '');
+        if (eventType === 'result') {
+            sawResult = true;
+            normalized.push(event);
+        }
+    }
+    if (!sawResult) {
+        if (errorMessage) {
+            normalized.push({ type: 'result', status: 'error', error: { message: errorMessage } });
+        }
+        else {
+            const inputTokens = usage ? numberFrom(usage.input_tokens) ?? numberFrom(usage.inputTokens) ?? 0 : 0;
+            const outputTokens = usage ? numberFrom(usage.output_tokens) ?? numberFrom(usage.outputTokens) ?? 0 : 0;
+            const stats = usage ? {
+                total_tokens: numberFrom(usage.total_tokens) ?? numberFrom(usage.totalTokens) ?? inputTokens + outputTokens,
+                input_tokens: inputTokens,
+                output_tokens: outputTokens,
+                cached: numberFrom(usage.cached) ?? numberFrom(usage.cached_tokens) ?? numberFrom(usage.cached_input_tokens) ?? numberFrom(usage.cachedTokens) ?? 0,
+            } : undefined;
+            normalized.push({ type: 'result', status: 'success', ...(stats ? { stats } : {}) });
+        }
+    }
+    return normalized.map((event) => JSON.stringify(event)).join('\n');
+}
+export class CodexRunner {
+    skillDispatchToolName = 'activate_skill';
+    linkedSkillsByWorktree = new Map();
+    async runPrompt(prompt, cwd, onLog, logPath, extraArgs = [], timeoutMs) {
+        return new Promise((resolve) => {
+            let stdout = '';
+            let stderr = '';
+            let resolved = false;
+            let timeout;
+            const linkedSkill = cwd ? this.linkedSkillsByWorktree.get(path.resolve(cwd)) : undefined;
+            const skillConfigArgs = linkedSkill
+                ? ['-c', `skills.config=[{path=${tomlString(linkedSkill.path)},enabled=true}]`]
+                : [];
+            const instructionArgs = linkedSkill
+                ? ['-c', `model_instructions_file=${tomlString(linkedSkill.instructionsPath)}`]
+                : [];
+            const args = [
+                'exec',
+                '--json',
+                '--cd', cwd ?? process.cwd(),
+                '--sandbox', 'workspace-write',
+                '-c', 'approval_policy="never"',
+                '--skip-git-repo-check',
+                '--ephemeral',
+                '--color', 'never',
+                ...skillConfigArgs,
+                ...instructionArgs,
+                ...extraArgs,
+                prompt,
+            ];
+            const spawnOptions = {
+                cwd,
+                env: { ...process.env, FORCE_COLOR: '0' },
+                stdio: ['ignore', 'pipe', 'pipe'],
+                detached: true,
+            };
+            function appendLog(chunk) {
+                if (!logPath)
+                    return;
+                try {
+                    fs.appendFileSync(logPath, chunk);
+                }
+                catch (err) {
+                    Logger.warn(`Failed to write Codex debug log at ${logPath}. Continuing. Reason: ${err}`);
+                }
+            }
+            function killProcessGroup() {
+                if (!child.pid)
+                    return;
+                try {
+                    process.kill(-child.pid, 'SIGKILL');
+                }
+                catch {
+                    // Process might have already exited.
+                }
+            }
+            const child = child_process.spawn('codex', args, spawnOptions);
+            appendLog(`--- Codex Execution Start: ${new Date().toISOString()} ---\n`);
+            appendLog(`Command: codex ${args.join(' ')}\n\n`);
+            if (timeoutMs && timeoutMs > 0) {
+                timeout = setTimeout(() => {
+                    if (resolved)
+                        return;
+                    resolved = true;
+                    killProcessGroup();
+                    const timeoutSec = timeoutMs / 1000;
+                    Logger.error(`\nCodex process timed out after ${timeoutSec} seconds.`);
+                    appendLog(`\n\n--- Codex process timed out ---\n${stderr}\n`);
+                    resolve({ error: `Process timeout exceeded (${timeoutSec} seconds)`, raw_output: stderr });
+                }, timeoutMs);
+            }
+            child.stdout?.on('data', (data) => {
+                const chunk = data.toString();
+                stdout += chunk;
+                appendLog(chunk);
+            });
+            child.stderr?.on('data', (data) => {
+                const chunk = data.toString();
+                stderr += chunk;
+                if (onLog) {
+                    const lines = chunk.split('\n').filter((line) => line.trim() !== '');
+                    if (lines.length > 0)
+                        onLog(lines[lines.length - 1]);
+                }
+            });
+            child.on('error', (err) => {
+                if (resolved)
+                    return;
+                resolved = true;
+                if (timeout)
+                    clearTimeout(timeout);
+                Logger.error(`Failed to start Codex CLI. Error: ${err.message}`);
+                appendLog(`\n\n--- Error starting Codex CLI ---\n${err.message}\n`);
+                resolve(null);
+            });
+            child.on('close', (code) => {
+                appendLog(`\n\n--- Codex exited with status ${code} ---\n`);
+                if (code !== 0 && stderr) {
+                    appendLog(`--- Stderr ---\n${stderr}\n--- End Stderr ---\n`);
+                }
+                if (resolved)
+                    return;
+                resolved = true;
+                if (timeout)
+                    clearTimeout(timeout);
+                if (!stdout.trim()) {
+                    resolve({ error: 'Empty output from Codex CLI', raw_output: stderr });
+                    return;
+                }
+                const normalized = normalizeCodexJsonl(stdout, linkedSkill?.name);
+                if (code !== 0) {
+                    resolve({
+                        error: `Codex CLI exited with status ${code}`,
+                        response: normalized,
+                        raw_output: `${normalized}\n--- CODEX STDOUT ---\n${stdout}\n--- STDERR ---\n${stderr}`,
+                    });
+                    return;
+                }
+                resolve({
+                    response: normalized,
+                    raw_output: `${normalized}\n--- CODEX STDOUT ---\n${stdout}\n--- STDERR ---\n${stderr}`,
+                });
+            });
+        });
+    }
+    applyRunnerConfig(evalConfigBaseDir, worktreePath) {
+        const src = path.join(evalConfigBaseDir, 'codex');
+        if (!fs.existsSync(src))
+            return;
+        const dst = path.join(worktreePath, '.codex');
+        fs.mkdirSync(dst, { recursive: true });
+        fs.cpSync(src, dst, { recursive: true, force: true });
+    }
+    async linkSkill(absoluteSkillPath, worktreePath) {
+        const skillName = readSkillName(absoluteSkillPath);
+        const localSkillsDir = path.join(worktreePath, '.codex', 'skills');
+        const symlinkPath = path.join(localSkillsDir, skillName);
+        fs.mkdirSync(localSkillsDir, { recursive: true });
+        if (fs.existsSync(symlinkPath)) {
+            fs.rmSync(symlinkPath, { recursive: true, force: true });
+        }
+        fs.symlinkSync(absoluteSkillPath, symlinkPath, 'dir');
+        const instructionsPath = path.join(worktreePath, '.codex', 'skill-eval-instructions.md');
+        fs.writeFileSync(instructionsPath, [
+            'You are running inside an automated Agent Skill evaluation.',
+            'Complete the user task end-to-end without asking for approval, waiting for confirmation, or stopping after a plan.',
+            'Use the configured skill when it applies. If you use a skill, read and follow its SKILL.md instructions.',
+            'Make any required edits directly in the current working directory.',
+            'Keep the final response concise.',
+            '',
+        ].join('\n'));
+        this.linkedSkillsByWorktree.set(path.resolve(worktreePath), {
+            name: skillName,
+            path: symlinkPath,
+            instructionsPath,
+        });
+    }
+}

package/dist/runners/index.js

@@ -1,2 +1,3 @@
 export { GeminiCliRunner } from './gemini-cli/index.js';
+export { CodexRunner } from './codex/index.js';
 export { RUNNER_REGISTRY, RunnerFactory, DEFAULT_AGENT } from './registry.js';

package/dist/runners/registry.js

@@ -1,6 +1,8 @@
 import { GeminiCliRunner } from './gemini-cli/index.js';
+import { CodexRunner } from './codex/index.js';
 export const RUNNER_REGISTRY = {
     'gemini-cli': { Runner: GeminiCliRunner, binary: 'gemini' },
+    codex: { Runner: CodexRunner, binary: 'codex' },
 };
 /** Default agent name used when none is specified on the CLI. */
 export const DEFAULT_AGENT = Object.keys(RUNNER_REGISTRY)[0];

package/package.json

@@ -1,7 +1,7 @@
 {
   "name": "@fede0089/skill-eval",
-  "version": "1.4.0",
-  "description": "CLI to evaluate agent skills triggering",
+  "version": "2.0.0",
+  "description": "CLI to evaluate agent skills triggering and functionality",
   "main": "dist/index.js",
   "bin": {
     "skill-eval": "dist/index.js"
@@ -18,8 +18,8 @@
     "prepare": "npm run build",
     "build": "tsc",
     "start": "node dist/index.js",
-    "test:trigger": "npm run build && node ./dist/index.js trigger --workspace . --skill ./mock-skill --debug",
-    "test:functional": "npm run build && node ./dist/index.js functional --workspace . --skill ./mock-skill --debug",
+    "test:trigger": "npm run build && node ./scripts/run-mock-eval.mjs trigger",
+    "test:functional": "npm run build && node ./scripts/run-mock-eval.mjs functional",
     "test:unit": "tsx --test \"tests/**/*.test.ts\""
   },
   "type": "module",