@plaited/agent-eval-harness 0.5.0

package/LICENSE

@@ -0,0 +1,15 @@
+ISC License
+
+Copyright (c) 2026 Plaited Labs
+
+Permission to use, copy, modify, and/or distribute this software for any
+purpose with or without fee is hereby granted, provided that the above
+copyright notice and this permission notice appear in all copies.
+
+THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES WITH
+REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF MERCHANTABILITY
+AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY SPECIAL, DIRECT,
+INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES WHATSOEVER RESULTING FROM
+LOSS OF USE, DATA OR PROFITS, WHETHER IN AN ACTION OF CONTRACT, NEGLIGENCE OR
+OTHER TORTIOUS ACTION, ARISING OUT OF OR IN CONNECTION WITH THE USE OR
+PERFORMANCE OF THIS SOFTWARE.

package/README.md

@@ -0,0 +1,273 @@
+# @plaited/agent-eval-harness
+
+[![npm version](https://img.shields.io/npm/v/@plaited/agent-eval-harness.svg)](https://www.npmjs.com/package/@plaited/agent-eval-harness)
+[![CI](https://github.com/plaited/agent-eval-harness/actions/workflows/ci.yml/badge.svg)](https://github.com/plaited/agent-eval-harness/actions/workflows/ci.yml)
+[![License: ISC](https://img.shields.io/badge/License-ISC-blue.svg)](https://opensource.org/licenses/ISC)
+
+CLI tool for capturing agent trajectories from headless CLI agents. Execute prompts, capture full trajectories (tools, thoughts, plans), and output structured JSONL for downstream scoring. Available as both a CLI tool and as installable skills for AI coding agents.
+
+## CLI Tool
+
+Use these tools directly via the CLI without installation:
+
+```bash
+# Using built-in headless adapter (recommended - no extra install needed)
+export ANTHROPIC_API_KEY=sk-...
+bunx @plaited/agent-eval-harness capture prompts.jsonl \
+  --schema ./schemas/claude-headless.json \
+  -o results.jsonl
+```
+
+**Prerequisite:** Set your API key. The harness works with any CLI agent that supports JSON output - just provide a schema describing how to interact with it:
+
+```bash
+export ANTHROPIC_API_KEY=sk-...   # For Claude
+export GEMINI_API_KEY=...         # For Gemini
+```
+
+Pre-built schemas are available in `.claude/skills/headless-adapters/schemas/` for Claude and Gemini.
+
+### Core Commands
+
+| Command | Description |
+|---------|-------------|
+| `capture <prompts> --schema <path>` | Trajectory capture (full JSONL) |
+| `trials <prompts> --schema <path>` | Multi-run with pass@k metrics |
+| `summarize <results>` | Derive compact views from results |
+| `calibrate <results>` | Sample failures for review |
+| `validate-refs <prompts>` | Check reference solutions |
+| `balance <prompts>` | Analyze test set coverage |
+| `schemas [name]` | Export JSON schemas |
+| `headless --schema <path>` | Schema-driven adapter for any CLI agent |
+
+### Pipeline Commands (Unix-style)
+
+| Command | Description |
+|---------|-------------|
+| `run <prompts> --schema <path>` | Execute prompts, output raw results |
+| `extract <raw> --schema <path>` | Parse raw output into trajectories |
+| `grade <results> --grader <path>` | Apply grader to extracted results |
+| `format <results> --style <style>` | Convert to markdown, csv, or jsonl |
+| `compare <run1> <run2>... --grader <path>` | Compare multiple runs |
+
+### Examples
+
+```bash
+# Capture trajectories using headless adapter (recommended)
+bunx @plaited/agent-eval-harness capture prompts.jsonl \
+  --schema ./schemas/claude-headless.json \
+  -o results.jsonl
+
+# Run trials for pass@k analysis with debug mode
+bunx @plaited/agent-eval-harness trials prompts.jsonl \
+  --schema ./schemas/claude-headless.json \
+  -k 5 --grader ./grader.ts --debug
+
+# Summarize results
+bunx @plaited/agent-eval-harness summarize results.jsonl -o summary.jsonl
+
+# Export schemas
+bunx @plaited/agent-eval-harness schemas CaptureResult --json
+
+# Pipeline workflow (Unix-style composition)
+cat prompts.jsonl | \
+  bunx @plaited/agent-eval-harness run -s ./schemas/claude-headless.json | \
+  bunx @plaited/agent-eval-harness extract -s ./schemas/claude-headless.json | \
+  bunx @plaited/agent-eval-harness grade -g ./grader.ts | \
+  bunx @plaited/agent-eval-harness format -f markdown > report.md
+
+# Compare multiple runs
+bunx @plaited/agent-eval-harness compare run1.jsonl run2.jsonl \
+  --grader ./compare-grader.ts -o comparison.jsonl
+```
+
+## Skills for AI Agents
+
+**Install skills** for use with AI coding agents:
+
+```bash
+curl -fsSL https://raw.githubusercontent.com/plaited/skills-installer/main/install.sh | bash -s -- --agent <agent-name> --project agent-eval-harness
+```
+
+Replace `<agent-name>` with your agent: `claude`, `cursor`, `copilot`, `opencode`, `amp`, `goose`, `factory`
+
+### Available Skills
+
+#### Agent Eval Harness
+
+CLI tool for capturing agent trajectories, optimized for TypeScript/JavaScript projects using Bun.
+
+**Core Commands:**
+
+| Command | Description |
+|---------|-------------|
+| `capture` | Execute prompts and capture full trajectories |
+| `trials` | Multi-run trials with pass@k/pass^k metrics |
+| `summarize` | Derive compact views from trajectory results |
+| `calibrate` | Sample failures for grader calibration |
+| `validate-refs` | Validate reference solutions against graders |
+| `balance` | Analyze test set coverage distribution |
+| `schemas` | Export Zod schemas as JSON Schema |
+
+**Pipeline Commands (Unix-style):**
+
+| Command | Description |
+|---------|-------------|
+| `run` | Execute prompts, output raw results |
+| `extract` | Parse raw output into trajectories |
+| `grade` | Apply grader to extracted results |
+| `format` | Convert to markdown, csv, or jsonl |
+| `compare` | Compare multiple runs |
+
+**Use cases:**
+- Capturing trajectories for downstream evaluation (Braintrust, custom scorers)
+- Generating training data (SFT/DPO) with full context
+- Building regression test fixtures for agent behavior
+- Comparing agent responses across configurations
+
+#### Headless Adapters
+
+Schema-driven adapters for headless CLI agent integration.
+
+**Commands:**
+
+| Command | Description |
+|---------|-------------|
+| `headless` | Schema-driven adapter for any CLI agent |
+
+**Use cases:**
+- Wrapping headless CLI agents with schema-driven adapter
+- Finding existing adapters for your agent
+- Creating new schemas for CLI agents
+
+## Input Format
+
+```jsonl
+{"id":"test-001","input":"Create a primary button","hint":"should contain <button>","metadata":{"category":"ui"}}
+{"id":"test-002","input":["Create a component","Now add tests"],"metadata":{"category":"multi-turn"}}
+```
+
+| Field | Required | Description |
+|-------|----------|-------------|
+| `id` | Yes | Unique identifier |
+| `input` | Yes | Single prompt (string) or conversation turns (string[]) |
+| `hint` | No | Grader context - what to look for |
+| `reference` | No | Reference solution (for validate-refs) |
+| `metadata` | No | Tags, category, difficulty for filtering |
+| `timeout` | No | Override default timeout for this prompt (ms) |
+
+## Output Format
+
+The harness outputs full trajectory JSONL (`CaptureResult` schema):
+
+```jsonl
+{
+  "id": "test-001",
+  "input": "Create a primary button",
+  "output": "Here's a button component...",
+  "hint": "should contain <button>",
+  "trajectory": [...],
+  "metadata": {"category": "ui", "trajectoryRichness": "full", "turnCount": 1},
+  "timing": {"start": 1234567890, "end": 1234567900, "total": 10},
+  "toolErrors": false,
+  "exitInfo": {"exitCode": 0},
+  "score": {"pass": true, "score": 1.0, "reasoning": "Contains hint"}
+}
+```
+
+Key fields:
+- `toolErrors`: Boolean indicating if any tool calls failed
+- `score`: Grader result (only if `--grader` provided)
+- `trajectory`: Full execution trace (thoughts, messages, tool calls, plans)
+- `metadata.trajectoryRichness`: `"full"` | `"messages-only"` | `"minimal"`
+- `exitInfo`: Process exit information (`exitCode`, `signal`, `timedOut`)
+- `timing.total`: End-to-end duration (ms)
+
+## Graders
+
+Graders score agent outputs. The harness supports two types:
+
+### TypeScript/JavaScript Graders
+
+Export a `grade` function:
+
+```typescript
+import type { Grader } from '@plaited/agent-eval-harness/schemas'
+
+export const grade: Grader = async ({ input, output, hint, trajectory }) => {
+  const pass = output.toLowerCase().includes(hint?.toLowerCase() ?? '')
+  return {
+    pass,
+    score: pass ? 1.0 : 0.0,
+    reasoning: pass ? 'Contains hint content' : 'Missing hint content'
+  }
+}
+```
+
+```bash
+agent-eval-harness capture prompts.jsonl --schema ./claude.json --grader ./grader.ts
+```
+
+### Polyglot Graders (Python, etc.)
+
+Any executable script using stdin/stdout JSON protocol:
+
+```python
+#!/usr/bin/env python3
+import json
+import sys
+
+data = json.load(sys.stdin)
+output = data["output"].lower()
+hint = (data.get("hint") or "").lower()
+
+pass_result = hint in output if hint else True
+print(json.dumps({
+    "pass": pass_result,
+    "score": 1.0 if pass_result else 0.0,
+    "reasoning": "Contains hint" if pass_result else "Missing hint"
+}))
+```
+
+```bash
+chmod +x grader.py
+agent-eval-harness capture prompts.jsonl --schema ./claude.json --grader ./grader.py
+```
+
+**Protocol:**
+- Input (stdin): `{"input": "...", "output": "...", "hint": "...", "trajectory": [...]}`
+- Output (stdout): `{"pass": true, "score": 1.0, "reasoning": "..."}`
+
+## Downstream Integration
+
+```bash
+# Filter failures
+cat results.jsonl | jq 'select(.score.pass == false)'
+
+# Extract tool usage patterns
+cat results.jsonl | jq '.trajectory[] | select(.type == "tool_call") | .name'
+
+# Use with your scoring pipeline
+cat results.jsonl | your-scoring-script.ts
+```
+
+## Development
+
+```bash
+bun install          # Install dependencies
+bun run check        # Type check + lint + format
+bun test             # Run unit tests
+
+# Run integration tests in Docker (requires API keys)
+ANTHROPIC_API_KEY=sk-... docker compose -f docker-compose.test.yml run --rm test
+```
+
+## Requirements
+
+- **Runtime:** Bun >= 1.2.9
+- **Schema:** JSON schema describing CLI agent interaction (see `.claude/skills/headless-adapters/schemas/`)
+- **API Key:** `ANTHROPIC_API_KEY` for Claude, `GEMINI_API_KEY` for Gemini
+
+## License
+
+ISC © [Plaited Labs](https://github.com/plaited)

package/bin/cli.ts

@@ -0,0 +1,162 @@
+#!/usr/bin/env bun
+
+/**
+ * Agent Eval Harness CLI - Agent evaluation toolkit.
+ *
+ * @remarks
+ * Router for harness commands. Thin wrapper that delegates to command modules.
+ *
+ * Commands:
+ * - capture: Core trajectory capture
+ * - trials: Multi-run pass@k/pass^k analysis
+ * - summarize: Derive compact views from results
+ * - calibrate: Sample failures for grader review
+ * - validate-refs: Check reference solutions
+ * - balance: Analyze test set coverage
+ * - schemas: Export JSON schemas for non-TS users
+ * - headless: Schema-driven adapter for any headless CLI agent
+ */
+
+import { balance } from '../src/commands/balance.ts'
+import { calibrate } from '../src/commands/calibrate.ts'
+import { capture } from '../src/commands/capture.ts'
+import { summarize } from '../src/commands/summarize.ts'
+import { trials } from '../src/commands/trials.ts'
+import { validateRefs } from '../src/commands/validate-refs.ts'
+import { headless } from '../src/headless.ts'
+import { compare, extract, format, grade, run } from '../src/pipeline.ts'
+import { schemasCli } from '../src/schemas/schemas-cli.ts'
+
+const [command, ...args] = Bun.argv.slice(2)
+
+const printHelp = () => {
+  // biome-ignore lint/suspicious/noConsole: CLI help output
+  console.log(`
+agent-eval-harness - CLI tool for agent evaluation
+
+Commands:
+  capture          Capture trajectories from CLI agents
+  trials           Run prompts multiple times for pass@k/pass^k metrics
+  summarize        Derive compact views from results
+  calibrate        Sample failures for grader review
+  validate-refs    Check reference solutions against grader
+  balance          Analyze test set coverage
+  schemas          Export JSON schemas for non-TypeScript users
+  headless         Schema-driven adapter for any headless CLI agent
+
+Pipeline Commands (Unix-style composable):
+  run              Execute prompts and output raw results
+  extract          Parse raw output into trajectories
+  grade            Apply grader to extracted results
+  format           Convert results to different output formats
+  compare          Compare multiple runs of the same prompts
+
+Run 'agent-eval-harness <command> --help' for command-specific help.
+
+Examples:
+  # Basic capture with schema
+  agent-eval-harness capture prompts.jsonl --schema claude.json -o results.jsonl
+
+  # With grader
+  agent-eval-harness capture prompts.jsonl -s claude.json --grader ./grader.ts -o results.jsonl
+
+  # Multi-run trials
+  agent-eval-harness trials prompts.jsonl -s claude.json -k 5 --grader ./grader.ts -o trials.jsonl
+
+  # Derive summary view
+  agent-eval-harness summarize results.jsonl -o summary.jsonl
+
+  # Pipeline workflow
+  cat prompts.jsonl | \\
+    agent-eval-harness run -s claude.json | \\
+    agent-eval-harness extract -s claude.json | \\
+    agent-eval-harness grade -g ./grader.ts | \\
+    agent-eval-harness format -f markdown > report.md
+
+  # Compare multiple runs
+  agent-eval-harness compare run1.jsonl run2.jsonl -g ./compare-grader.ts
+
+Documentation: https://github.com/plaited/agent-eval-harness
+`)
+}
+
+const main = async () => {
+  switch (command) {
+    case 'capture':
+      await capture(args)
+      break
+
+    case 'trials':
+      await trials(args)
+      break
+
+    case 'summarize':
+      await summarize(args)
+      break
+
+    case 'calibrate':
+      await calibrate(args)
+      break
+
+    case 'validate-refs':
+      await validateRefs(args)
+      break
+
+    case 'balance':
+      await balance(args)
+      break
+
+    case 'schemas':
+      await schemasCli(args)
+      break
+
+    case 'headless':
+      await headless(args)
+      break
+
+    // Pipeline commands
+    case 'run':
+      await run(args)
+      break
+
+    case 'extract':
+      await extract(args)
+      break
+
+    case 'grade':
+      await grade(args)
+      break
+
+    case 'format':
+      await format(args)
+      break
+
+    case 'compare':
+      await compare(args)
+      break
+
+    case '-h':
+    case '--help':
+    case undefined:
+      printHelp()
+      break
+
+    case '-v':
+    case '--version': {
+      const { version } = await import('../package.json')
+      // biome-ignore lint/suspicious/noConsole: CLI version output
+      console.log(version)
+      break
+    }
+
+    default:
+      console.error(`Unknown command: ${command}`)
+      console.error("Run 'agent-eval-harness --help' for usage")
+      process.exit(1)
+  }
+}
+
+main().catch((error) => {
+  console.error('Error:', error instanceof Error ? error.message : error)
+  process.exit(1)
+})