@fede0089/skill-eval 2.0.0 → 3.0.1

package/README.md

@@ -2,11 +2,18 @@
 
 A CLI tool for evaluating Agent Skills locally. Tests whether your skill triggers reliably and produces the right output, using an LLM as the judge.
 
-## Why skill evals?
+## What you can test
 
-Skills are instructions that change how the agent behaves. But a single successful run isn't enough to trust one — agents are non-deterministic, and a good isolated result can be exactly that: an isolated case.
+skill-eval ships two commands, each targeting a different failure mode:
 
-Skill evals let you turn that intuition into evidence: run several comparable tasks with the skill, measure them against the same criteria, and optionally compare against a baseline or historical skill branches.
+- **Triggering** (`skill-eval trigger`) — checks whether the agent actually decides to invoke the skill in the right context. A skill that never gets triggered cannot help, no matter how good its instructions are.
+- **Functional correctness** (`skill-eval functional`) — checks whether the actions the agent takes while the skill is active match your expectations. An LLM judge grades each transcript against the expectation list you provide.
+
+## Why run skill evals
+
+- **Avoid regression** — it is very common that, while iterating to make a skill handle a new case, it stops solving cases it used to handle. Evals make every iteration measurable, so you can tell whether a change adds value without silently subtracting it elsewhere.
+- **Validate against a baseline** — sometimes an agent solves a task better using its general capabilities than using a specific skill (and even if it works today, model upgrades can shift that balance). Comparing against the no-skill baseline (`--compare-baseline`) or past skill versions (`--compare-ref`) tells you whether the skill is really pulling its weight.
+- **Statistical confidence** — LLMs are non-deterministic, so a single passing run is not evidence. Running the same expectations N times produces a pass rate (pass@k) that turns "it feels like it works" into a number you can defend.
 
 ## How it works
 
@@ -37,7 +44,7 @@ For each eval prompt, skill-eval spins up parallel agent processes with the curr
 
 ## Installation
 
-**Requirements:** Node.js, and the agent CLI you want to evaluate (e.g. `gemini` or `codex`) installed and on `$PATH`.
+**Requirements:** Node.js, and the agent CLI you want to evaluate (e.g. `gemini`, `codex`, or `claude`) installed and on `$PATH`.
 
 ### Run without installing
 
@@ -91,6 +98,7 @@ Supported runners:
 
 - `gemini-cli` (default)
 - `codex`
+- `claude-code`
 
 ### Skill directory structure
 
@@ -102,8 +110,10 @@ my-skill/
     └── config/                     # runner configuration (optional but often needed)
         ├── gemini-cli/             # copied to <worktree>/.gemini/ before each trial
         │   └── settings.json
-        └── codex/                  # copied to <worktree>/.codex/ before each trial
-            └── config.toml
+        ├── codex/                  # copied to <worktree>/.codex/ before each trial
+        │   └── config.toml
+        └── claude-code/            # copied to <worktree>/.claude/ before each trial
+            └── settings.json
 ```
 
 All `.json` files in `evals/` are loaded and merged into a single suite — you can split them by feature or regression category.
@@ -141,16 +151,34 @@ All `.json` files in `evals/` are loaded and merged into a single suite — you
 
 skill-eval runs the agent headlessly — stdin is closed, there is no terminal. If the agent encounters a tool that requires interactive approval, it will either fail immediately or hang until the trial timeout kills it.
 
-Each runner configures its own non-interactive mode. For example, Gemini CLI uses `--approval-mode auto_edit`, while Codex uses `codex exec --json --sandbox workspace-write -c approval_policy="never"`. If your skill needs to run shell commands, read environment variables, make network calls, or use any other tool category, refer to that runner's permission model.
+Each runner configures its own non-interactive mode. For example, Gemini CLI uses `--approval-mode auto_edit`, Codex uses `codex exec --json --sandbox workspace-write -c approval_policy="never"`, and Claude Code uses `claude -p --output-format stream-json --permission-mode bypassPermissions`. If your skill needs to run shell commands, read environment variables, make network calls, or use any other tool category, refer to that runner's permission model.
 
 **Solution:** place a config file inside your skill at `evals/config/<runner>/`. Before every trial, skill-eval automatically copies that directory into the agent's config location inside the isolated worktree:
 
 ```
-evals/config/gemini-cli/  →  <worktree>/.gemini/
-evals/config/codex/       →  <worktree>/.codex/
+evals/config/gemini-cli/   →  <worktree>/.gemini/
+evals/config/codex/        →  <worktree>/.codex/
+evals/config/claude-code/  →  <worktree>/.claude/
+```
+
+Use this to ship both settings and policies alongside your evals. Anything inside `evals/config/<runner>/` is dropped verbatim into the runner's config directory, so you can use the runner's full configuration surface — not just `settings.json`.
+
+### Gemini CLI example
+
+Gemini CLI reads both `settings.json` and any `*.toml` rule files under `policies/`. Drop them inside `evals/config/gemini-cli/` and skill-eval will copy them into `<worktree>/.gemini/` before each trial:
+
+```
+my-skill/
+└── evals/
+    └── config/
+        └── gemini-cli/
+            ├── settings.json
+            └── policies/
+                ├── allow-activate-skill.toml
+                └── allow-tools.toml
 ```
 
-Use this to ship both settings and policies alongside your evals. For Gemini CLI, for example, you can use `settings.json` to configure tool permissions and approval policies so that every tool your skill relies on runs without prompting:
+`settings.json` holds general configuration (telemetry, model, etc.):
 
 ```json
 {
@@ -158,10 +186,42 @@ Use this to ship both settings and policies alongside your evals. For Gemini CLI
 }
 ```
 
-Refer to your runner's documentation for the full list of available settings and policy keys.
+Policies whitelist specific tools so they run without prompting. For example, to always allow the `activate_skill` dispatch tool in non-interactive mode:
+
+```toml
+# evals/config/gemini-cli/policies/allow-activate-skill.toml
+[[rule]]
+toolName    = "activate_skill"
+decision    = "allow"
+priority    = 100
+interactive = false
+```
+
+Refer to your runner's documentation for the full set of settings and policy keys (Codex uses `config.toml`, Claude Code uses `settings.json`).
 
 > This config only applies inside the temporary worktree created for each trial. Your real workspace config is never touched.
 
+## Reports
+
+Each run writes to `.project-skill-evals/runs/<timestamp>/` and includes per-trial logs, the raw eval JSON, and a self-contained HTML report you can open in any browser. The report shows pass@k aggregates per eval, lets you expand each trial, and color-codes triggering vs. functional outcomes.
+
+A published sample report is available at [fede0089.github.io/skill-eval/sample-report.html](https://fede0089.github.io/skill-eval/sample-report.html),generated from this project root with:
+
+```sh
+skill-eval functional --workspace . --skill mock-skill --trials 2 --compare-baseline --debug claude-code
+```
+
+![Sample HTML report](docs/sample-report.png)
+
+### Debug logs
+
+When a trial misbehaves, pass `-v` / `--debug` to capture the full transcripts to disk. Each trial writes a `task_<id>_<variant>_trial_<n>.log` file inside the run directory with two sections appended in order:
+
+- `# SECTION: <MODE> AGENT RUN` — the initial prompt sent to the agent and its raw streamed response.
+- `# SECTION: <MODE> JUDGE RUN` — the prompt sent to the LLM judge and its verdict (only present for `functional` runs; `trigger` is graded programmatically and produces no judge section).
+
+Without `--debug` these files are not written, so reach for the flag when you need to see exactly what the agent — or the judge — saw.
+
 ## Try it out
 
 This repo includes a `mock-skill/` directory — a complete, working example of a license-generator skill with trigger and functional evals. Run it directly with:
@@ -170,12 +230,12 @@ This repo includes a `mock-skill/` directory — a complete, working example of
 npm run test:unit        # run the unit test suite
 npm run test:trigger     # trigger evaluation against mock-skill
 npm run test:functional  # functional evaluation against mock-skill
-npm run test:trigger -- codex     # run trigger evals with Codex
-npm run test:functional -- codex  # run functional evals with Codex
+npm run test:trigger -- codex            # run trigger evals with Codex
+npm run test:functional -- codex         # run functional evals with Codex
+npm run test:trigger -- claude-code      # run trigger evals with Claude Code
+npm run test:functional -- claude-code   # run functional evals with Claude Code
 ```
 
-Results are saved to `.project-skill-evals/runs/<timestamp>/` with logs, raw eval JSONs, and an HTML report.
-
 ## Extending
 
 ### Adding a new agent runner

package/dist/runners/claude-code/index.js

@@ -0,0 +1 @@
+export { ClaudeCodeRunner } from './runner.js';

package/dist/runners/claude-code/runner.js

@@ -0,0 +1,279 @@
+import * as fs from 'fs';
+import * as path from 'path';
+import child_process from 'child_process';
+import { Logger } from '../../utils/logger.js';
+function isRecord(value) {
+    return typeof value === 'object' && value !== null;
+}
+function readSkillName(skillPath) {
+    const skillMd = path.join(skillPath, 'SKILL.md');
+    if (!fs.existsSync(skillMd))
+        return path.basename(skillPath);
+    const content = fs.readFileSync(skillMd, 'utf-8');
+    const frontmatter = content.match(/^---\n([\s\S]*?)\n---/);
+    if (!frontmatter)
+        return path.basename(skillPath);
+    const nameLine = frontmatter[1].split('\n').find((line) => line.trim().startsWith('name:'));
+    const name = nameLine?.replace(/^name:\s*/, '').trim().replace(/^["']|["']$/g, '');
+    return name || path.basename(skillPath);
+}
+function numberFrom(value) {
+    return typeof value === 'number' ? value : undefined;
+}
+function getAssistantContent(event) {
+    const eventType = String(event.type ?? '');
+    if (eventType !== 'assistant')
+        return [];
+    const message = event.message;
+    if (!isRecord(message))
+        return [];
+    const content = message.content;
+    if (!Array.isArray(content))
+        return [];
+    return content.filter(isRecord);
+}
+function getUserContent(event) {
+    const eventType = String(event.type ?? '');
+    if (eventType !== 'user')
+        return [];
+    const message = event.message;
+    if (!isRecord(message))
+        return [];
+    const content = message.content;
+    if (!Array.isArray(content))
+        return [];
+    return content.filter(isRecord);
+}
+/**
+ * Converts Claude Code stream-json events into the small NDJSON contract used
+ * by the rest of the evaluator. The normalizer is intentionally tolerant: the
+ * Claude Code event stream is richer than the evaluator needs and may grow
+ * over time. Only fields the grader/judge consume are mapped.
+ */
+export function normalizeClaudeJsonl(output) {
+    const normalized = [];
+    let sawResult = false;
+    for (const line of output.split('\n')) {
+        const trimmed = line.trim();
+        if (!trimmed)
+            continue;
+        let event;
+        try {
+            event = JSON.parse(trimmed);
+        }
+        catch {
+            continue;
+        }
+        for (const item of getAssistantContent(event)) {
+            const itemType = String(item.type ?? '');
+            if (itemType === 'text' && typeof item.text === 'string' && item.text.trim()) {
+                normalized.push({ type: 'message', role: 'assistant', content: item.text });
+            }
+            else if (itemType === 'tool_use') {
+                const toolName = String(item.name ?? '');
+                const toolId = String(item.id ?? `claude-tool-${normalized.length + 1}`);
+                const input = isRecord(item.input) ? item.input : {};
+                // The internal NDJSON contract expects parameters.name to carry the skill
+                // identifier when the dispatch tool is invoked. Claude Code's Skill tool
+                // uses `input.skill` for that — remap so TriggerGrader can match it.
+                const parameters = { ...input };
+                if (toolName === 'Skill' && typeof input.skill === 'string') {
+                    parameters.name = input.skill;
+                }
+                normalized.push({
+                    type: 'tool_use',
+                    tool_id: toolId,
+                    tool_name: toolName,
+                    parameters,
+                });
+            }
+        }
+        for (const item of getUserContent(event)) {
+            const itemType = String(item.type ?? '');
+            if (itemType !== 'tool_result')
+                continue;
+            const toolId = String(item.tool_use_id ?? '');
+            if (!toolId)
+                continue;
+            const isError = item.is_error === true;
+            // Claude Code attaches a tool_use_result sibling on the parent event with
+            // a `success` boolean for built-in tools (e.g. Skill). Prefer that when present.
+            const parent = event.tool_use_result;
+            const parentSuccess = isRecord(parent) ? parent.success : undefined;
+            const status = isError === true || parentSuccess === false ? 'error' : 'success';
+            normalized.push({ type: 'tool_result', tool_id: toolId, status });
+        }
+        const eventType = String(event.type ?? '');
+        if (eventType === 'result') {
+            sawResult = true;
+            const isError = event.is_error === true;
+            const usage = isRecord(event.usage) ? event.usage : undefined;
+            const inputTokens = usage ? numberFrom(usage.input_tokens) ?? 0 : 0;
+            const outputTokens = usage ? numberFrom(usage.output_tokens) ?? 0 : 0;
+            const cachedTokens = usage
+                ? numberFrom(usage.cache_read_input_tokens)
+                    ?? numberFrom(usage.cached_input_tokens)
+                    ?? 0
+                : 0;
+            const stats = usage
+                ? {
+                    total_tokens: inputTokens + outputTokens,
+                    input_tokens: inputTokens,
+                    output_tokens: outputTokens,
+                    cached: cachedTokens,
+                }
+                : undefined;
+            if (isError) {
+                const message = typeof event.result === 'string' && event.result.trim()
+                    ? event.result
+                    : `Claude Code run failed (subtype: ${String(event.subtype ?? 'unknown')})`;
+                normalized.push({ type: 'result', status: 'error', error: { message }, ...(stats ? { stats } : {}) });
+            }
+            else {
+                const response = typeof event.result === 'string' ? event.result : undefined;
+                normalized.push({
+                    type: 'result',
+                    status: 'success',
+                    ...(response ? { response } : {}),
+                    ...(stats ? { stats } : {}),
+                });
+            }
+        }
+    }
+    if (!sawResult) {
+        normalized.push({ type: 'result', status: 'error', error: { message: 'Claude Code produced no result event' } });
+    }
+    return normalized.map((event) => JSON.stringify(event)).join('\n');
+}
+export class ClaudeCodeRunner {
+    skillDispatchToolName = 'Skill';
+    linkedSkillsByWorktree = new Map();
+    async runPrompt(prompt, cwd, onLog, logPath, extraArgs = [], timeoutMs) {
+        return new Promise((resolve) => {
+            let stdout = '';
+            let stderr = '';
+            let resolved = false;
+            let timeout;
+            const args = [
+                '-p', prompt,
+                '--output-format', 'stream-json',
+                '--verbose',
+                '--permission-mode', 'bypassPermissions',
+                '--no-session-persistence',
+                ...extraArgs,
+            ];
+            const spawnOptions = {
+                cwd,
+                env: { ...process.env, FORCE_COLOR: '0' },
+                stdio: ['ignore', 'pipe', 'pipe'],
+                detached: true,
+            };
+            function appendLog(chunk) {
+                if (!logPath)
+                    return;
+                try {
+                    fs.appendFileSync(logPath, chunk);
+                }
+                catch (err) {
+                    Logger.warn(`Failed to write Claude Code debug log at ${logPath}. Continuing. Reason: ${err}`);
+                }
+            }
+            function killProcessGroup() {
+                if (!child.pid)
+                    return;
+                try {
+                    process.kill(-child.pid, 'SIGKILL');
+                }
+                catch {
+                    // Process might have already exited.
+                }
+            }
+            const child = child_process.spawn('claude', args, spawnOptions);
+            appendLog(`--- Claude Code Execution Start: ${new Date().toISOString()} ---\n`);
+            appendLog(`Command: claude ${args.join(' ')}\n\n`);
+            if (timeoutMs && timeoutMs > 0) {
+                timeout = setTimeout(() => {
+                    if (resolved)
+                        return;
+                    resolved = true;
+                    killProcessGroup();
+                    const timeoutSec = timeoutMs / 1000;
+                    Logger.error(`\nClaude Code process timed out after ${timeoutSec} seconds.`);
+                    appendLog(`\n\n--- Claude Code process timed out ---\n${stderr}\n`);
+                    resolve({ error: `Process timeout exceeded (${timeoutSec} seconds)`, raw_output: stderr });
+                }, timeoutMs);
+            }
+            child.stdout?.on('data', (data) => {
+                const chunk = data.toString();
+                stdout += chunk;
+                appendLog(chunk);
+            });
+            child.stderr?.on('data', (data) => {
+                const chunk = data.toString();
+                stderr += chunk;
+                if (onLog) {
+                    const lines = chunk.split('\n').filter((line) => line.trim() !== '');
+                    if (lines.length > 0)
+                        onLog(lines[lines.length - 1]);
+                }
+            });
+            child.on('error', (err) => {
+                if (resolved)
+                    return;
+                resolved = true;
+                if (timeout)
+                    clearTimeout(timeout);
+                Logger.error(`Failed to start Claude Code CLI. Error: ${err.message}`);
+                appendLog(`\n\n--- Error starting Claude Code CLI ---\n${err.message}\n`);
+                resolve(null);
+            });
+            child.on('close', (code) => {
+                appendLog(`\n\n--- Claude Code exited with status ${code} ---\n`);
+                if (code !== 0 && stderr) {
+                    appendLog(`--- Stderr ---\n${stderr}\n--- End Stderr ---\n`);
+                }
+                if (resolved)
+                    return;
+                resolved = true;
+                if (timeout)
+                    clearTimeout(timeout);
+                if (!stdout.trim()) {
+                    resolve({ error: 'Empty output from Claude Code CLI', raw_output: stderr });
+                    return;
+                }
+                const normalized = normalizeClaudeJsonl(stdout);
+                if (code !== 0) {
+                    resolve({
+                        error: `Claude Code CLI exited with status ${code}`,
+                        response: normalized,
+                        raw_output: `${normalized}\n--- CLAUDE CODE STDOUT ---\n${stdout}\n--- STDERR ---\n${stderr}`,
+                    });
+                    return;
+                }
+                resolve({
+                    response: normalized,
+                    raw_output: `${normalized}\n--- CLAUDE CODE STDOUT ---\n${stdout}\n--- STDERR ---\n${stderr}`,
+                });
+            });
+        });
+    }
+    applyRunnerConfig(evalConfigBaseDir, worktreePath) {
+        const src = path.join(evalConfigBaseDir, 'claude-code');
+        if (!fs.existsSync(src))
+            return;
+        const dst = path.join(worktreePath, '.claude');
+        fs.mkdirSync(dst, { recursive: true });
+        fs.cpSync(src, dst, { recursive: true, force: true });
+    }
+    async linkSkill(absoluteSkillPath, worktreePath) {
+        const skillName = readSkillName(absoluteSkillPath);
+        const localSkillsDir = path.join(worktreePath, '.claude', 'skills');
+        const symlinkPath = path.join(localSkillsDir, skillName);
+        fs.mkdirSync(localSkillsDir, { recursive: true });
+        if (fs.existsSync(symlinkPath)) {
+            fs.rmSync(symlinkPath, { recursive: true, force: true });
+        }
+        fs.symlinkSync(absoluteSkillPath, symlinkPath, 'dir');
+        this.linkedSkillsByWorktree.set(path.resolve(worktreePath), { name: skillName });
+    }
+}

package/dist/runners/index.js

@@ -1,3 +1,4 @@
 export { GeminiCliRunner } from './gemini-cli/index.js';
 export { CodexRunner } from './codex/index.js';
+export { ClaudeCodeRunner } from './claude-code/index.js';
 export { RUNNER_REGISTRY, RunnerFactory, DEFAULT_AGENT } from './registry.js';

package/dist/runners/registry.js

@@ -1,8 +1,10 @@
 import { GeminiCliRunner } from './gemini-cli/index.js';
 import { CodexRunner } from './codex/index.js';
+import { ClaudeCodeRunner } from './claude-code/index.js';
 export const RUNNER_REGISTRY = {
     'gemini-cli': { Runner: GeminiCliRunner, binary: 'gemini' },
     codex: { Runner: CodexRunner, binary: 'codex' },
+    'claude-code': { Runner: ClaudeCodeRunner, binary: 'claude' },
 };
 /** Default agent name used when none is specified on the CLI. */
 export const DEFAULT_AGENT = Object.keys(RUNNER_REGISTRY)[0];

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@fede0089/skill-eval",
-  "version": "2.0.0",
+  "version": "3.0.1",
   "description": "CLI to evaluate agent skills triggering and functionality",
   "main": "dist/index.js",
   "bin": {