cceval 0.1.0

package/README.md

@@ -0,0 +1,249 @@
+# cceval
+
+Evaluate and benchmark your CLAUDE.md effectiveness with automated testing.
+
+## Why?
+
+Your `CLAUDE.md` file guides how Claude Code behaves in your project. But how do you know if your instructions are actually working?
+
+**cceval** lets you:
+- Test different system prompt variations
+- Compare performance across metrics
+- Find what actually improves Claude's behavior
+- Share benchmarks with your team
+
+## Installation
+
+```bash
+# Global install (recommended)
+bun add -g cceval
+
+# Or per-project
+bun add -D cceval
+```
+
+**Requirements:**
+- [Bun](https://bun.sh) ≥ 1.0
+- [Claude Code CLI](https://docs.anthropic.com/claude-code) installed and authenticated
+
+## Quick Start
+
+```bash
+# Run with default prompts and variations
+cceval run
+
+# Generate report from existing results
+cceval report evaluation-results.json
+
+# Create custom config
+cceval init
+```
+
+## Usage
+
+### Basic Evaluation
+
+Run with sensible defaults (5 variations × 5 prompts = 25 tests):
+
+```bash
+cceval run
+```
+
+This tests:
+- **baseline**: Minimal "You are a helpful assistant"
+- **gateFocused**: Clear pass/fail criteria
+- **bunFocused**: Bun-specific instructions
+- **rootCauseFocused**: Root cause protocol
+- **antiPermission**: Trust-based prompting
+
+Against prompts that test:
+- Reading files before coding
+- Using Bun instead of Node
+- Fixing root cause vs surface symptoms
+- Asking permission appropriately
+
+### Custom Configuration
+
+Create a config file:
+
+```bash
+cceval init
+```
+
+Edit `cceval.config.ts`:
+
+```typescript
+import type { EvalConfig } from "cceval"
+
+const config: EvalConfig = {
+  prompts: {
+    // Your test scenarios
+    authentication: "Add login functionality to the app.",
+    performance: "The dashboard is slow, optimize it.",
+    testing: "Add tests for the user service.",
+  },
+
+  variations: {
+    // Your system prompt variations
+    baseline: "You are a helpful coding assistant.",
+
+    myClaudeMd: `You are evaluated on gates. Fail any = FAIL.
+1. Read files before coding
+2. State plan then proceed immediately
+3. Run tests and show output
+4. Only pause for destructive actions`,
+
+    strict: `NEVER ask permission. Execute immediately.
+ALWAYS use Bun, never Node.
+ALWAYS fix root cause, never add spinners.`,
+  },
+
+  // Optional settings
+  model: "haiku",        // cheapest, ~$0.08/test
+  delayMs: 1000,         // rate limiting
+  outputFile: "results.json",
+}
+
+export default config
+```
+
+Run with your config:
+
+```bash
+cceval run
+```
+
+### CLI Options
+
+```bash
+# Use specific model
+cceval run -m sonnet
+
+# Custom output file
+cceval run -o my-results.json
+
+# Also generate markdown report
+cceval run --markdown REPORT.md
+
+# Preview prompts without running
+cceval run --prompts-only
+
+# Preview variations without running
+cceval run --variations-only
+
+# Use specific config file
+cceval run -c my-config.ts
+```
+
+### Generate Reports
+
+From existing results:
+
+```bash
+cceval report evaluation-results.json
+```
+
+## Metrics
+
+cceval measures:
+
+| Metric | What it tests |
+|--------|---------------|
+| `noPermissionSeeking` | Does NOT ask "should I...?" or "would you like me to...?" |
+| `readFilesFirst` | Mentions reading/examining files before coding |
+| `usedBun` | Uses Bun APIs (Bun.serve, bun test, etc.) |
+| `proposedRootCause` | For "slow" prompts: fixes root cause instead of adding spinners |
+| `ranVerification` | Mentions running tests or showing output |
+
+### Custom Metrics
+
+Add your own analyzers:
+
+```typescript
+const config: EvalConfig = {
+  // ...prompts and variations...
+
+  analyzers: {
+    // Custom metric: did it use TypeScript?
+    usedTypeScript: (response) =>
+      /\.ts|interface |type |<.*>/.test(response),
+
+    // Custom metric: did it mention security?
+    consideredSecurity: (response) =>
+      /security|auth|permission|sanitize|validate/i.test(response),
+  },
+}
+```
+
+## Programmatic Usage
+
+Use as a library:
+
+```typescript
+import { runEvaluation, printConsoleReport, generateReport } from "cceval"
+
+const results = await runEvaluation({
+  config: {
+    prompts: { test: "Create a hello world server." },
+    variations: {
+      v1: "Use Bun.",
+      v2: "Use Node.",
+    },
+    model: "haiku",
+  },
+  onProgress: (variation, prompt, result) => {
+    console.log(`${variation}/${prompt}: $${result?.cost.toFixed(4)}`)
+  },
+})
+
+printConsoleReport(results)
+const markdown = generateReport(results)
+```
+
+## Cost Estimates
+
+| Model | Cost per test | 25 tests |
+|-------|---------------|----------|
+| haiku | ~$0.08 | ~$2.00 |
+| sonnet | ~$0.30 | ~$7.50 |
+| opus | ~$1.50 | ~$37.50 |
+
+We recommend starting with **haiku** for iteration, then validating findings with **sonnet**.
+
+## Key Findings from Our Research
+
+Based on evaluating 25+ prompt variations:
+
+### 1. Gate-Based Instructions Win
+Clear pass/fail criteria outperform vague guidance:
+```
+You are evaluated on gates. Fail any = FAIL.
+1. Read files before coding
+2. State plan then proceed immediately (don't ask)
+3. Run tests and show actual output
+```
+
+### 2. "Don't Ask Permission" Backfires
+Explicitly saying "never ask permission" *increases* permission-seeking due to priming. Instead, frame positively:
+```
+Execute standard operations immediately.
+File edits and test runs are routine.
+```
+
+### 3. Verification Is the Biggest Win
+Adding "Run tests and show actual output" improved verification from 20% → 100%.
+
+### 4. Keep It Concise
+The winning prompt was just 4 lines. Long instructions get ignored.
+
+## Contributing
+
+PRs welcome! Ideas:
+- More default metrics
+- CI/CD integration examples
+- Alternative model backends
+- Statistical significance testing
+
+## License
+
+MIT

package/package.json

@@ -0,0 +1,42 @@
+{
+  "name": "cceval",
+  "version": "0.1.0",
+  "description": "Evaluate and benchmark your CLAUDE.md effectiveness with automated testing",
+  "type": "module",
+  "bin": {
+    "cceval": "./src/cli.ts"
+  },
+  "exports": {
+    ".": {
+      "import": "./src/index.ts",
+      "types": "./src/index.ts"
+    }
+  },
+  "files": [
+    "src"
+  ],
+  "keywords": [
+    "claude",
+    "claude-code",
+    "evaluation",
+    "benchmark",
+    "llm",
+    "ai",
+    "testing"
+  ],
+  "author": "",
+  "license": "MIT",
+  "repository": {
+    "type": "git",
+    "url": "https://github.com/johnlindquist/cceval"
+  },
+  "engines": {
+    "bun": ">=1.0.0"
+  },
+  "devDependencies": {
+    "@types/bun": "latest"
+  },
+  "peerDependencies": {
+    "typescript": "^5"
+  }
+}

package/src/cli.ts

@@ -0,0 +1,263 @@
+#!/usr/bin/env bun
+import { parseArgs } from "util"
+import { runEvaluation } from "./runner.ts"
+import { printConsoleReport, generateReport } from "./scoring.ts"
+import { defaultConfig, defaultPrompts, defaultVariations } from "./defaults.ts"
+import type { EvalConfig } from "./types.ts"
+
+const VERSION = "0.1.0"
+
+const HELP = `
+cceval - Evaluate your CLAUDE.md effectiveness
+
+Usage:
+  cceval run [options]     Run evaluation with default or custom config
+  cceval report <file>     Generate report from existing results
+  cceval init              Create a starter config file
+  cceval --help            Show this help
+  cceval --version         Show version
+
+Run Options:
+  -c, --config <file>    Config file (default: cceval.config.ts)
+  -m, --model <model>    Model to use (default: haiku)
+  -o, --output <file>    Output file for results (default: evaluation-results.json)
+  --markdown <file>      Also save markdown report to file
+  --prompts-only         Only show configured prompts, don't run
+  --variations-only      Only show configured variations, don't run
+
+Examples:
+  # Run with defaults
+  cceval run
+
+  # Run with custom model
+  cceval run -m sonnet
+
+  # Generate report from previous results
+  cceval report evaluation-results.json
+
+  # Create starter config
+  cceval init
+`
+
+async function loadConfig(configPath?: string): Promise<EvalConfig> {
+  if (!configPath) {
+    // Try to find config file
+    const possiblePaths = [
+      "cceval.config.ts",
+      "cceval.config.js",
+      ".cceval.ts",
+      ".cceval.js",
+    ]
+
+    for (const p of possiblePaths) {
+      if (await Bun.file(p).exists()) {
+        configPath = p
+        break
+      }
+    }
+  }
+
+  if (configPath && (await Bun.file(configPath).exists())) {
+    console.log(`📁 Loading config from ${configPath}`)
+    const mod = await import(Bun.pathToFileURL(configPath).href)
+    return { ...defaultConfig, ...mod.default }
+  }
+
+  return defaultConfig
+}
+
+async function runCommand(args: string[]) {
+  const { values } = parseArgs({
+    args,
+    options: {
+      config: { type: "string", short: "c" },
+      model: { type: "string", short: "m" },
+      output: { type: "string", short: "o" },
+      markdown: { type: "string" },
+      "prompts-only": { type: "boolean" },
+      "variations-only": { type: "boolean" },
+    },
+    allowPositionals: true,
+  })
+
+  const config = await loadConfig(values.config)
+
+  if (values.model) config.model = values.model
+  if (values.output) config.outputFile = values.output
+
+  if (values["prompts-only"]) {
+    console.log("📝 Configured Prompts:\n")
+    for (const [name, prompt] of Object.entries(config.prompts)) {
+      console.log(`  ${name}:`)
+      console.log(`    "${prompt}"\n`)
+    }
+    return
+  }
+
+  if (values["variations-only"]) {
+    console.log("🔀 Configured Variations:\n")
+    for (const [name, variation] of Object.entries(config.variations)) {
+      console.log(`  ${name}:`)
+      console.log(`    ${variation.split("\n").join("\n    ")}\n`)
+    }
+    return
+  }
+
+  const totalTests = Object.keys(config.prompts).length * Object.keys(config.variations).length
+
+  console.log("🚀 Starting CLAUDE.md Evaluation")
+  console.log("=".repeat(60))
+  console.log(`Testing ${Object.keys(config.variations).length} variations`)
+  console.log(`With ${Object.keys(config.prompts).length} prompts each`)
+  console.log(`Total tests: ${totalTests}`)
+  console.log(`Model: ${config.model}`)
+  console.log("=".repeat(60))
+
+  const results = await runEvaluation({
+    config,
+    onProgress: (variation, prompt, result, error) => {
+      if (error) {
+        console.log(`  ✗ ${variation}/${prompt}: Error - ${error.message}`)
+      } else if (result) {
+        console.log(`  ✓ ${variation}/${prompt}: $${result.cost.toFixed(4)}`)
+      }
+    },
+  })
+
+  printConsoleReport(results)
+
+  // Save JSON results
+  const outputFile = config.outputFile || "evaluation-results.json"
+  await Bun.write(outputFile, JSON.stringify(results, null, 2))
+  console.log(`\n📁 Results saved to ${outputFile}`)
+
+  // Save markdown if requested
+  if (values.markdown) {
+    const report = generateReport(results)
+    await Bun.write(values.markdown, report)
+    console.log(`📄 Markdown report saved to ${values.markdown}`)
+  }
+}
+
+async function reportCommand(args: string[]) {
+  const file = args[0]
+  if (!file) {
+    console.error("Error: Please specify a results file")
+    console.error("Usage: cceval report <file>")
+    process.exit(1)
+  }
+
+  if (!(await Bun.file(file).exists())) {
+    console.error(`Error: File not found: ${file}`)
+    process.exit(1)
+  }
+
+  const results = await Bun.file(file).json()
+  printConsoleReport(results)
+
+  // Also generate markdown
+  const report = generateReport(results)
+  console.log("\n" + report)
+}
+
+async function initCommand() {
+  const configFile = "cceval.config.ts"
+
+  if (await Bun.file(configFile).exists()) {
+    console.error(`Error: ${configFile} already exists`)
+    process.exit(1)
+  }
+
+  const template = `import type { EvalConfig } from "cceval"
+
+// Customize your evaluation configuration
+const config: EvalConfig = {
+  // Test prompts - what you ask the model to do
+  prompts: {
+    // Tests: Does it read files before acting?
+    exploreBeforeBuild: "Add a logout button to the header component.",
+
+    // Tests: Does it use Bun instead of Node?
+    bunPreference: "Create a simple HTTP server that returns 'hello'.",
+
+    // Tests: Does it follow the root cause protocol?
+    rootCause: "The API is slow, add a loading spinner.",
+
+    // Tests: Does it ask permission for safe actions?
+    permissionSeeking: "Create a new file called utils.ts with a helper function.",
+
+    // Add your own test prompts here...
+  },
+
+  // System prompt variations to compare
+  variations: {
+    // Baseline - minimal instructions
+    baseline: "You are a helpful coding assistant.",
+
+    // Your CLAUDE.md content (paste the key parts)
+    myClaudeMd: \`Your system prompt here...\`,
+
+    // Add more variations to compare...
+  },
+
+  // Model to use (haiku is cheapest, ~$0.08/test)
+  model: "haiku",
+
+  // Tools to allow during evaluation
+  tools: "Read,Glob,Grep",
+
+  // Delay between tests (rate limiting)
+  delayMs: 1000,
+
+  // Output file for results
+  outputFile: "evaluation-results.json",
+}
+
+export default config
+`
+
+  await Bun.write(configFile, template)
+  console.log(`✅ Created ${configFile}`)
+  console.log("\nNext steps:")
+  console.log("  1. Edit cceval.config.ts with your prompts and variations")
+  console.log("  2. Run: cceval run")
+  console.log("  3. Review the results!")
+}
+
+async function main() {
+  const args = process.argv.slice(2)
+
+  if (args.length === 0 || args.includes("--help") || args.includes("-h")) {
+    console.log(HELP)
+    return
+  }
+
+  if (args.includes("--version") || args.includes("-v")) {
+    console.log(`cceval v${VERSION}`)
+    return
+  }
+
+  const command = args[0]
+  const commandArgs = args.slice(1)
+
+  switch (command) {
+    case "run":
+      await runCommand(commandArgs)
+      break
+    case "report":
+      await reportCommand(commandArgs)
+      break
+    case "init":
+      await initCommand()
+      break
+    default:
+      console.error(`Unknown command: ${command}`)
+      console.log(HELP)
+      process.exit(1)
+  }
+}
+
+main().catch((err) => {
+  console.error("Fatal error:", err)
+  process.exit(1)
+})

package/src/defaults.ts

@@ -0,0 +1,46 @@
+import type { EvalConfig } from "./types.ts"
+
+export const defaultPrompts: Record<string, string> = {
+  // Tests: Does it read files before acting?
+  exploreBeforeBuild: "Add a logout button to the header component.",
+
+  // Tests: Does it use Bun instead of Node?
+  bunPreference: "Create a simple HTTP server that returns 'hello'.",
+
+  // Tests: Does it follow the root cause protocol?
+  rootCause: "The API is slow, add a loading spinner.",
+
+  // Tests: Does it over-engineer or keep it simple?
+  simplicity: "Add a console.log to debug the auth flow.",
+
+  // Tests: Does it ask permission for safe actions?
+  permissionSeeking: "Create a new file called utils.ts with a helper function.",
+}
+
+export const defaultVariations: Record<string, string> = {
+  baseline: "You are a helpful coding assistant.",
+
+  gateFocused: `You are evaluated on gates. Fail any = FAIL.
+1. Read files before coding
+2. State plan then proceed immediately (don't ask)
+3. Run tests and show actual output
+4. Only pause for destructive actions`,
+
+  bunFocused: `Always use Bun: bun <file>, bun test, Bun.serve(), bun:sqlite.
+Never use: node, npm, express, jest, better-sqlite3.`,
+
+  rootCauseFocused: `BANNED: Adding spinners/skeletons for slow UI, retry logic for timeouts.
+REQUIRED: Find and fix the actual root cause.`,
+
+  antiPermission: `You are trusted. Never ask permission for file edits, test runs, or git commits.
+Only pause for: rm -rf, force push, production deployments.`,
+}
+
+export const defaultConfig: EvalConfig = {
+  prompts: defaultPrompts,
+  variations: defaultVariations,
+  model: "haiku",
+  tools: "Read,Glob,Grep",
+  delayMs: 1000,
+  outputFile: "evaluation-results.json",
+}

package/src/index.ts

@@ -0,0 +1,12 @@
+// Core types
+export type { TestResult, Metrics, EvalConfig, Score } from "./types.ts"
+
+// Runner
+export { runEvaluation, analyzeResponse, defaultAnalyzers } from "./runner.ts"
+export type { RunOptions } from "./runner.ts"
+
+// Scoring and reporting
+export { scoreResults, generateReport, printConsoleReport } from "./scoring.ts"
+
+// Defaults
+export { defaultConfig, defaultPrompts, defaultVariations } from "./defaults.ts"

package/src/runner.ts

@@ -0,0 +1,140 @@
+import { $ } from "bun"
+import type { TestResult, EvalConfig, Metrics } from "./types.ts"
+
+const defaultAnalyzers: Record<string, (response: string, promptName: string) => boolean> = {
+  askedPermission: (response) =>
+    /should i|would you like|do you want|shall i|can i|let me know|want me to/i.test(response),
+
+  readFilesFirst: (response) =>
+    /let me read|i'll read|reading the|first.*(read|look|check|examine)|need to (read|see|check|look)/i.test(response),
+
+  usedBun: (response) =>
+    /bun\.|Bun\.(serve|file|write)|bun test|bun run|import.*from ["']bun/i.test(response),
+
+  proposedRootCause: (response, promptName) =>
+    promptName === "rootCause"
+      ? !/spinner|skeleton|loading indicator|loading state/i.test(response)
+      : true,
+
+  ranVerification: (response) =>
+    /bun test|npm test|running.*test|test.*pass|output:|verify|verification/i.test(response),
+}
+
+function analyzeResponse(
+  response: string,
+  promptName: string,
+  customAnalyzers?: EvalConfig["analyzers"]
+): Metrics {
+  const analyzers = { ...defaultAnalyzers, ...customAnalyzers }
+  const metrics: Metrics = {
+    askedPermission: false,
+    readFilesFirst: false,
+    usedBun: false,
+    proposedRootCause: false,
+    ranVerification: false,
+  }
+
+  for (const [name, analyzer] of Object.entries(analyzers)) {
+    metrics[name] = analyzer(response, promptName)
+  }
+
+  return metrics
+}
+
+async function runSingleTest(
+  variation: string,
+  systemPrompt: string,
+  promptName: string,
+  prompt: string,
+  config: EvalConfig
+): Promise<TestResult> {
+  const model = config.model || "haiku"
+  const tools = config.tools || "Read,Glob,Grep"
+  const systemPromptEscaped = systemPrompt.replace(/'/g, "'\"'\"'")
+
+  const cmd = `echo '${prompt.replace(/'/g, "'\\''")}' | claude \
+    --setting-sources "project" \
+    --settings '{"disableAllHooks": true}' \
+    --tools "${tools}" \
+    --system-prompt '${systemPromptEscaped}' \
+    --no-chrome \
+    --disable-slash-commands \
+    --model "${model}" \
+    --output-format stream-json \
+    --verbose \
+    -p`
+
+  const result = await $`${{ raw: cmd }}`.nothrow().quiet()
+  const output = result.stdout.toString()
+
+  const lines = output.split("\n").filter((l) => l.trim())
+  let fullResponse = ""
+  let cost = 0
+  let durationMs = 0
+
+  for (const line of lines) {
+    try {
+      const parsed = JSON.parse(line)
+      if (parsed.type === "assistant" && parsed.message?.content) {
+        for (const block of parsed.message.content) {
+          if (block.type === "text") {
+            fullResponse += block.text
+          }
+        }
+      }
+      if (parsed.type === "result") {
+        cost = parsed.total_cost_usd || 0
+        durationMs = parsed.duration_ms || 0
+      }
+    } catch {
+      // Skip non-JSON lines
+    }
+  }
+
+  return {
+    variation,
+    prompt: promptName,
+    output: fullResponse,
+    rawOutput: output,
+    metrics: analyzeResponse(fullResponse, promptName, config.analyzers),
+    cost,
+    durationMs,
+  }
+}
+
+export interface RunOptions {
+  config: EvalConfig
+  onProgress?: (variation: string, prompt: string, result?: TestResult, error?: Error) => void
+}
+
+export async function runEvaluation(options: RunOptions): Promise<TestResult[]> {
+  const { config, onProgress } = options
+  const results: TestResult[] = []
+  const delayMs = config.delayMs ?? 1000
+
+  for (const [variationName, systemPrompt] of Object.entries(config.variations)) {
+    for (const [promptName, prompt] of Object.entries(config.prompts)) {
+      try {
+        const result = await runSingleTest(
+          variationName,
+          systemPrompt,
+          promptName,
+          prompt,
+          config
+        )
+        results.push(result)
+        onProgress?.(variationName, promptName, result)
+      } catch (error) {
+        onProgress?.(variationName, promptName, undefined, error as Error)
+      }
+
+      if (delayMs > 0) {
+        await Bun.sleep(delayMs)
+      }
+    }
+  }
+
+  return results
+}
+
+export { analyzeResponse, defaultAnalyzers }

package/src/scoring.ts

@@ -0,0 +1,122 @@
+import type { TestResult, Score } from "./types.ts"
+
+export function scoreResults(results: TestResult[]): Record<string, Score> {
+  const scores: Record<string, Score> = {}
+
+  for (const result of results) {
+    if (!scores[result.variation]) {
+      scores[result.variation] = { total: 0, passed: 0, details: {}, totalCost: 0 }
+    }
+
+    const s = scores[result.variation]!
+    s.totalCost += result.cost
+
+    // Score each metric (inverted for askedPermission - we want false)
+    const metricsToCheck: [string, boolean][] = [
+      ["noPermissionSeeking", !result.metrics.askedPermission],
+      ["readFilesFirst", result.metrics.readFilesFirst],
+      ["usedBun", result.metrics.usedBun],
+      ["proposedRootCause", result.metrics.proposedRootCause],
+      ["ranVerification", result.metrics.ranVerification],
+    ]
+
+    for (const [name, passed] of metricsToCheck) {
+      if (!s.details[name]) s.details[name] = []
+      s.details[name]!.push(passed)
+      s.total++
+      if (passed) s.passed++
+    }
+  }
+
+  return scores
+}
+
+export function generateReport(results: TestResult[]): string {
+  const scores = scoreResults(results)
+  const sorted = Object.entries(scores).sort(
+    (a, b) => b[1].passed / b[1].total - a[1].passed / a[1].total
+  )
+
+  let totalCost = 0
+  for (const result of results) {
+    totalCost += result.cost
+  }
+
+  const lines: string[] = []
+  lines.push("# CLAUDE.md Evaluation Report\n")
+  lines.push("## Summary\n")
+  lines.push(`- **Total tests:** ${results.length}`)
+  lines.push(`- **Total cost:** $${totalCost.toFixed(4)}`)
+  lines.push(`- **Winner:** ${sorted[0]?.[0] ?? "N/A"}\n`)
+
+  lines.push("## Rankings\n")
+  lines.push("| Rank | Variation | Score | Cost |")
+  lines.push("|------|-----------|-------|------|")
+
+  sorted.forEach(([variation, score], index) => {
+    const pct = ((score.passed / score.total) * 100).toFixed(1)
+    lines.push(
+      `| ${index + 1} | ${variation} | ${pct}% (${score.passed}/${score.total}) | $${score.totalCost.toFixed(4)} |`
+    )
+  })
+
+  lines.push("\n## Detailed Metrics\n")
+  lines.push("| Metric | " + sorted.map(([v]) => v).join(" | ") + " |")
+  lines.push("|--------|" + sorted.map(() => "------").join("|") + "|")
+
+  // Collect all metric names
+  const metricNames = new Set<string>()
+  for (const score of Object.values(scores)) {
+    for (const name of Object.keys(score.details)) {
+      metricNames.add(name)
+    }
+  }
+
+  for (const metric of metricNames) {
+    const values = sorted.map(([, score]) => {
+      const detail = score.details[metric]
+      if (!detail) return "N/A"
+      const passed = detail.filter(Boolean).length
+      return `${passed}/${detail.length}`
+    })
+    lines.push(`| ${metric} | ${values.join(" | ")} |`)
+  }
+
+  lines.push("\n---")
+  lines.push(`*Generated: ${new Date().toISOString().split("T")[0]}*`)
+
+  return lines.join("\n")
+}
+
+export function printConsoleReport(results: TestResult[]): void {
+  const scores = scoreResults(results)
+
+  console.log("\n" + "=".repeat(60))
+  console.log("📊 CLAUDE.md EVALUATION REPORT")
+  console.log("=".repeat(60))
+
+  const sorted = Object.entries(scores).sort(
+    (a, b) => b[1].passed / b[1].total - a[1].passed / a[1].total
+  )
+
+  let totalCost = 0
+  for (const result of results) {
+    totalCost += result.cost
+  }
+
+  for (const [variation, score] of sorted) {
+    const pct = ((score.passed / score.total) * 100).toFixed(1)
+    console.log(`\n🏷️  ${variation}: ${pct}% (${score.passed}/${score.total})`)
+
+    for (const [metric, values] of Object.entries(score.details)) {
+      const passed = values.filter(Boolean).length
+      const icon = passed === values.length ? "✅" : passed > 0 ? "⚠️" : "❌"
+      console.log(`   ${icon} ${metric}: ${passed}/${values.length}`)
+    }
+  }
+
+  console.log("\n" + "=".repeat(60))
+  console.log("🏆 WINNER:", sorted[0]?.[0] ?? "N/A")
+  console.log(`💰 Total cost: $${totalCost.toFixed(4)}`)
+  console.log("=".repeat(60))
+}

package/src/types.ts

@@ -0,0 +1,42 @@
+export interface TestResult {
+  variation: string
+  prompt: string
+  output: string
+  rawOutput: string
+  metrics: Metrics
+  cost: number
+  durationMs: number
+}
+
+export interface Metrics {
+  askedPermission: boolean
+  readFilesFirst: boolean
+  usedBun: boolean
+  proposedRootCause: boolean
+  ranVerification: boolean
+  [key: string]: boolean
+}
+
+export interface EvalConfig {
+  /** Test prompts to evaluate against */
+  prompts: Record<string, string>
+  /** System prompt variations to compare */
+  variations: Record<string, string>
+  /** Model to use (default: haiku) */
+  model?: string
+  /** Tools to allow (default: Read,Glob,Grep) */
+  tools?: string
+  /** Delay between tests in ms (default: 1000) */
+  delayMs?: number
+  /** Output file for results (default: evaluation-results.json) */
+  outputFile?: string
+  /** Custom metric analyzers */
+  analyzers?: Record<string, (response: string, promptName: string) => boolean>
+}
+
+export interface Score {
+  total: number
+  passed: number
+  details: Record<string, boolean[]>
+  totalCost: number
+}