@plaited/acp-harness 0.2.5 → 0.3.1

package/.claude/skills/acp-harness/SKILL.md

@@ -1,310 +0,0 @@
----
-name: acp-harness
-description: CLI tool for capturing agent trajectories. Execute prompts against ACP-compatible agents, capture full trajectories (tools, thoughts, plans), and output structured JSONL for downstream scoring.
-compatibility: Bun >= 1.2.9
----
-
-# ACP Harness
-
-## Purpose
-
-CLI tool for capturing trajectories from ACP-compatible agents, optimized for TypeScript/JavaScript projects using Bun.
-
-**The harness captures. You score.**
-
-| Harness Provides | You Provide |
-|------------------|-------------|
-| Prompt execution against ACP agents | Scoring logic (Braintrust, custom scripts) |
-| Full trajectory capture (thoughts, tools, plans) | Pass/fail determination |
-| Structured JSONL output | LLM-as-judge prompts |
-| Reproducible execution environment | CI integration, golden file comparison |
-
-**Use this when:**
-- Capturing trajectories for downstream evaluation
-- Generating training data (SFT/DPO) with full context
-- Building regression test fixtures for agent behavior
-- Comparing agent responses across configurations
-
-## Installation
-
-```bash
-# Run without installing (recommended for CI)
-bunx @plaited/acp-harness prompts.jsonl -o results.jsonl
-
-# Or install globally for repeated use
-bun add -g @plaited/acp-harness
-acp-harness prompts.jsonl -o results.jsonl
-
-# Or add as project dependency
-bun add @plaited/acp-harness
-```
-
-**Note:** Examples below use `acp-harness` (the command available after global install). Replace with `bunx @plaited/acp-harness` if not installed globally.
-
-## Capture Workflow
-
-```mermaid
-flowchart LR
-    Prompts["prompts.jsonl"] --> Harness["acp-harness"]
-    Agent["ACP Agent"] --> Harness
-    Harness -->|"JSONL"| Output["trajectories"]
-    Output --> Scoring["Your scoring logic"]
-    Scoring --> Decision["Informed choices"]
-```
-
-The harness is a **capture layer** - it executes prompts and records trajectories. Scoring happens in your codebase.
-
-| Use Case | Harness Captures | You Build |
-|----------|------------------|-----------|
-| **Agent comparison** | Same prompts → multiple agents → trajectories | Scoring pipeline (Braintrust, custom) |
-| **Tool comparison** | Trajectory with tool/skill attribution | Diff analysis, preference data |
-| **Training data** | Structured I/O with tool calls, plans, thoughts | SFT/DPO formatting |
-| **Regression testing** | Deterministic prompt → trajectory capture | Golden file comparison, CI assertions |
-
-### Example: Comparing Built-in vs Skill
-
-```bash
-# Run same prompt with built-in tool
-acp-harness prompts.jsonl \
-  --cmd "bunx claude-code-acp" \
-  -o results-builtin.jsonl
-
-# Run same prompt with custom skill installed
-acp-harness prompts.jsonl \
-  --cmd "bunx claude-code-acp" \
-  --cwd /project/with/typescript-lsp-skill \
-  -o results-skill.jsonl
-
-# Compare trajectories - which used better tools? faster? more accurate?
-diff <(jq '.toolCalls' results-builtin.jsonl) <(jq '.toolCalls' results-skill.jsonl)
-```
-
-## Execution Environment
-
-**Recommendation:** Run the harness in Docker containers for consistent, isolated execution.
-
-```bash
-# Build and run with Docker Compose
-docker compose -f docker-compose.acp.yml run --rm acp-harness
-
-# Or build directly
-docker build -f Dockerfile.acp -t acp-harness .
-docker run --rm -e ANTHROPIC_API_KEY acp-harness
-```
-
-Docker provides:
-- Consistent environment across local and CI
-- Filesystem isolation without app-level sandboxing
-- Reproducible results for training data generation
-
-See [assets/](assets/) for example container configurations:
-- `Dockerfile.acp` - Base container with Bun and git
-- `docker-compose.acp.yml` - Compose file with volume mounts for results
-
-## Non-Goals
-
-This harness is optimized for TypeScript/JavaScript projects using Bun. It is **not** designed for:
-
-- **Python projects** - Use [SWE-bench](https://github.com/SWE-bench/SWE-bench), [Braintrust Python SDK](https://www.braintrust.dev/)
-- **Academic model benchmarking** - Use [EleutherAI lm-evaluation-harness](https://github.com/EleutherAI/lm-evaluation-harness)
-- **IDE integrations** - Use Copilot Evaluation Harness
-- **SaaS observability** - Use Braintrust, Langfuse platforms directly
-
-## Quick Reference
-
-| Resource | Description |
-|----------|-------------|
-| `bunx @plaited/acp-harness` | Execute prompts against agent, capture trajectories |
-| [output-formats.md](references/output-formats.md) | JSONL schemas, format options |
-| [downstream.md](references/downstream.md) | Integration patterns (Braintrust, jq, custom scorers) |
-
-## Output Pipeline
-
-```mermaid
-flowchart LR
-    Prompts["prompts.jsonl"] --> Harness["acp-harness"]
-    Agent["ACP Agent"] --> Harness
-    Harness --> Summary["summary.jsonl"]
-    Harness --> Full["results.md + results.full.jsonl"]
-    Summary --> Your["Your scoring code"]
-    Full --> Your
-```
-
-1. **Prepare** - Create `prompts.jsonl` with test cases
-2. **Execute** - Run harness against target agent
-3. **Capture** - Trajectories streamed to output files
-4. **Score** - Pipe output to your scoring logic (Braintrust, jq, LLM-as-judge)
-
-## Harness Script
-
-### Basic Usage
-
-```bash
-acp-harness <prompts.jsonl> --cmd <cmd> [options]
-```
-
-### Arguments
-
-| Flag | Description | Default |
-|------|-------------|---------|
-| `prompts.jsonl` | Input file with prompts to execute | Required |
-| `--cmd, --command` | ACP agent command (e.g., `bunx claude-code-acp`, `bun ./adapter.ts`) | `"claude-code-acp"` |
-| `-o, --output` | Output file/path | stdout |
-| `-c, --cwd` | Working directory for agent | current |
-| `-t, --timeout` | Request timeout in ms | `60000` |
-| `-f, --format` | Output format: `summary`, `judge` | `summary` |
-| `--progress` | Show progress to stderr | false |
-| `--append` | Append to output file | false |
-| `--mcp-server` | MCP server config JSON (repeatable) | none |
-
-### Examples
-
-```bash
-# Using the default claude-code-acp adapter
-acp-harness prompts.jsonl -o results.jsonl
-
-# Using bunx to run an adapter
-acp-harness prompts.jsonl --cmd "bunx claude-code-acp" -o results.jsonl
-
-# Using a local adapter script (great for custom adapters in same repo)
-acp-harness prompts.jsonl --cmd "bun ./my-adapter.ts" -o results.jsonl
-
-# Judge format - creates two files for downstream scoring
-acp-harness prompts.jsonl --format judge -o results
-# Creates: results.md (summary with step IDs) + results.full.jsonl (complete trajectory)
-
-# With MCP server (stdio transport)
-acp-harness prompts.jsonl \
-  --mcp-server '{"type":"stdio","name":"fs","command":["mcp-filesystem","/data"]}'
-
-# With MCP server (HTTP transport)
-acp-harness prompts.jsonl \
-  --mcp-server '{"type":"http","name":"api","url":"http://localhost:3000"}'
-
-# Stream with progress
-acp-harness prompts.jsonl --progress -o results.jsonl
-```
-
-## Input Format
-
-Each line in `prompts.jsonl`:
-
-```jsonl
-{"id":"test-001","input":"Create a primary button","expected":"should contain <button>","metadata":{"category":"ui"}}
-{"id":"test-002","input":"Write a function for form validation","metadata":{"category":"logic"}}
-```
-
-| Field | Required | Description |
-|-------|----------|-------------|
-| `id` | Yes | Unique identifier |
-| `input` | Yes | Prompt text for the agent |
-| `expected` | No | Expected output (for downstream scoring) |
-| `metadata` | No | Tags, category, difficulty for filtering |
-| `timeout` | No | Override default timeout for this prompt |
-
-## Output Formats
-
-### Summary Format (default)
-
-Minimal JSONL for quick metrics and analysis:
-
-```jsonl
-{"id":"test-001","input":"Create a button","output":"I created...","toolCalls":["Write"],"status":"passed","duration":1234}
-```
-
-### Judge Format (two-tier)
-
-Creates two files optimized for downstream LLM-as-judge scoring:
-
-**`<output>.md`** - Markdown summary with step IDs and code previews:
-
-```markdown
-## Capture Record: test-001
-
-**Input:** Create a primary button
-
-**Trajectory:**
-1. [THOUGHT] I'll create a styled button... [->test-001-step-1]
-2. [TOOL:Write] -> completed (234ms) [->test-001-step-2]
-   File: src/button.tsx (847 chars)
-   ```tsx
-   import { css } from 'some-css-lib'
-
-   type ButtonProps = {
-     label: string
-
-   // ... 30 lines omitted ...
-
-   export const Button = ({ label }: ButtonProps) => (
-     <button className={styles.btn}>{label}</button>
-   )
-   ```
-3. [MESSAGE] I created the button... [->test-001-step-3]
-
-**Output:** I created the button with primary styling.
-**Metadata:** category=ui, agent=claude-code-acp
-**Status:** passed
-**Duration:** 1234ms
-
----
-```
-
-**`<output>.full.jsonl`** - Complete trajectory with step IDs for correlation:
-
-```jsonl
-{"id":"test-001","input":"...","output":"...","trajectory":[{"type":"thought","content":"...","timestamp":100,"stepId":"test-001-step-1"},{"type":"tool_call","name":"Write","status":"completed","input":{...},"output":{...},"duration":234,"stepId":"test-001-step-2"}],...}
-```
-
-**Usage patterns by judge context window:**
-
-| Judge Model | Strategy |
-|-------------|----------|
-| Gemini (1M+ tokens) | Feed `results.full.jsonl` directly |
-| Claude/GPT-4 (128-200k) | Use `results.full.jsonl` for most runs |
-| Smaller models | Use `results.md`, retrieve specific steps by ID as needed |
-
-## Downstream Integration
-
-The harness outputs standard JSONL that pipes to any tool:
-
-```bash
-# Filter with jq
-cat results.jsonl | jq 'select(.metadata.category == "ui")'
-
-# Count tool usage
-cat results.jsonl | jq -s 'map(.toolCalls | length) | add'
-
-# Feed full trajectory to Gemini (large context)
-cat results.full.jsonl | your-gemini-judge.ts
-```
-
-See [downstream.md](references/downstream.md) for integration patterns with Braintrust, Gemini, and custom scorers.
-
-## Capture Targets
-
-| Target | How to Capture |
-|--------|----------------|
-| **Agent capability** | Direct prompts, capture trajectory for analysis |
-| **Skills** | Set `--cwd` to project with skill, capture skill-specific behavior |
-| **MCP Servers** | Use `--mcp-server` flag, capture tool usage in trajectory |
-
-### Capturing Skill Behavior
-
-```bash
-bunx @plaited/acp-harness skill-prompts.jsonl \
-  --cwd /project/with/skill \
-  -o results.jsonl
-```
-
-### Capturing MCP Server Usage
-
-```bash
-bunx @plaited/acp-harness mcp-prompts.jsonl \
-  --mcp-server '{"type":"stdio","name":"fs","command":["mcp-filesystem"]}' \
-  -o results.jsonl
-```
-
-## Related
-
-- **[@agentclientprotocol/sdk](https://www.npmjs.com/package/@agentclientprotocol/sdk)** - ACP SDK for programmatic access
-- **[@zed-industries/claude-code-acp](https://www.npmjs.com/package/@zed-industries/claude-code-acp)** - Claude Code ACP adapter

package/.claude/skills/acp-harness/assets/Dockerfile.acp

@@ -1,25 +0,0 @@
-# ACP Harness Docker Configuration
-#
-# Example Dockerfile for running ACP evaluations in an isolated container.
-# Copy this to your project and customize as needed.
-#
-# Usage:
-#   docker build -f Dockerfile.acp -t acp-harness .
-#   docker run --rm -e ANTHROPIC_API_KEY acp-harness bunx @plaited/acp-harness prompts.jsonl
-
-FROM oven/bun:1.2.9
-
-# Install git (required for some agent operations)
-RUN apt-get update && apt-get install -y git && rm -rf /var/lib/apt/lists/*
-
-WORKDIR /app
-
-# Copy package files first for better layer caching
-COPY package.json bun.lock* ./
-RUN bun install --frozen-lockfile
-
-# Copy source files
-COPY . .
-
-# Default command - override with your harness invocation
-CMD ["bun", "test"]

package/.claude/skills/acp-harness/assets/docker-compose.acp.yml

@@ -1,19 +0,0 @@
-# ACP Harness Docker Compose Configuration
-#
-# Example docker-compose for running ACP evaluations.
-# Copy this to your project and customize as needed.
-#
-# Usage:
-#   ANTHROPIC_API_KEY=sk-... docker compose -f docker-compose.acp.yml run --rm acp-harness
-
-services:
-  acp-harness:
-    build:
-      context: .
-      dockerfile: Dockerfile.acp
-    environment:
-      - ANTHROPIC_API_KEY
-    volumes:
-      # Mount output directory to persist results
-      - ./results:/app/results
-    command: ["bunx", "@plaited/acp-harness", "prompts.jsonl", "-o", "results/output.jsonl"]

package/.claude/skills/acp-harness/references/downstream.md

@@ -1,288 +0,0 @@
-# Downstream Integration
-
-Patterns for piping harness output to analysis tools.
-
-## Loading Results
-
-Both output formats use JSONL (newline-delimited JSON):
-
-```typescript
-// TypeScript pattern (validated in tests)
-const parseResults = (jsonl: string) =>
-  jsonl.trim().split('\n').map((line) => JSON.parse(line))
-
-// Load from file
-const results = parseResults(await Bun.file('results.jsonl').text())
-```
-
-## jq Analysis
-
-Summary JSONL is designed for quick analysis with `jq`:
-
-```bash
-# Calculate average duration
-cat results.jsonl | jq -s 'map(.duration) | add / length'
-
-# Count tool usage
-cat results.jsonl | jq -s 'map(.toolCalls) | flatten | group_by(.) | map({tool: .[0], count: length})'
-
-# Filter by status
-cat results.jsonl | jq 'select(.status == "failed")'
-
-# Pass rate
-cat results.jsonl | jq -s 'map(select(.status == "passed")) | length as $p | length as $t | "\($p)/\($t) passed"'
-
-# Group by category
-cat results.jsonl | jq -s 'group_by(.metadata.category) | map({category: .[0].metadata.category, count: length})'
-
-# Find slowest runs
-cat results.jsonl | jq -s 'sort_by(-.duration) | .[0:5] | map({id, duration})'
-```
-
-## TypeScript Analysis Patterns
-
-These patterns are validated by tests in `bin/tests/cli.spec.ts`:
-
-### Filter by Status
-
-```typescript
-const failed = results.filter((r) => r.status === 'failed')
-const passed = results.filter((r) => r.status === 'passed')
-const passRate = passed.length / results.length
-```
-
-### Filter by Tool Usage
-
-```typescript
-// Find runs that used Write tool
-const withWrite = results.filter((r) => r.toolCalls.includes('Write'))
-
-// Find runs that used multiple tools
-const multiTool = results.filter((r) => r.toolCalls.length > 1)
-```
-
-### Filter by Duration
-
-```typescript
-// Slow runs (> 2 seconds)
-const slow = results.filter((r) => r.duration > 2000)
-
-// Find top 5 slowest
-const slowest = [...results].sort((a, b) => b.duration - a.duration).slice(0, 5)
-```
-
-### Filter by Metadata
-
-```typescript
-// Filter by category
-const uiResults = results.filter((r) => r.metadata.category === 'ui')
-
-// Group and count by category
-const grouped = results.reduce<Record<string, number>>((acc, r) => {
-  const cat = r.metadata.category as string
-  acc[cat] = (acc[cat] ?? 0) + 1
-  return acc
-}, {})
-```
-
-### Count Tool Usage
-
-```typescript
-const allTools = results.flatMap((r) => r.toolCalls)
-const toolCounts = allTools.reduce<Record<string, number>>((acc, tool) => {
-  acc[tool] = (acc[tool] ?? 0) + 1
-  return acc
-}, {})
-```
-
-### Deduplicate by ID
-
-```typescript
-// Keep latest occurrence when merging multiple runs
-const byId = new Map<string, unknown>()
-for (const result of results) {
-  byId.set(result.id, result)
-}
-const deduped = Array.from(byId.values())
-```
-
-## Step-Level Retrieval
-
-For judge format, correlate markdown step IDs with full JSONL:
-
-```typescript
-// Load both files
-const markdown = await Bun.file('results.md').text()
-const fullResults = parseResults(await Bun.file('results.full.jsonl').text())
-
-// Build step index
-const stepIndex = new Map<string, unknown>()
-for (const result of fullResults) {
-  for (const step of result.trajectory) {
-    stepIndex.set(step.stepId, step)
-  }
-}
-
-// Retrieve full step by ID (from markdown [→stepId])
-const stepId = 'test-001-step-2'
-const fullStep = stepIndex.get(stepId) as { name: string; input: unknown }
-console.log('Tool name:', fullStep.name)
-console.log('Full input:', fullStep.input)
-```
-
-## Extract Tool Calls from Trajectory
-
-```typescript
-const toolCalls = result.trajectory.filter((s) => s.type === 'tool_call')
-const toolNames = toolCalls.map((t) => t.name)
-```
-
-## Timing Information
-
-```typescript
-const result = results[0]
-const duration = result.timing.end - result.timing.start
-const timeToFirstResponse = result.timing.firstResponse // ms after start
-```
-
-## LLM-as-Judge
-
-### Large Context Models (Gemini 1M+)
-
-Feed full trajectory directly:
-
-```typescript
-import { GoogleGenerativeAI } from '@google/generative-ai'
-
-const genAI = new GoogleGenerativeAI(process.env.GOOGLE_API_KEY!)
-const model = genAI.getGenerativeModel({ model: 'gemini-2.5-pro' })
-
-const results = parseResults(await Bun.file('results.full.jsonl').text())
-
-const prompt = `
-Evaluate these agent trajectories for code quality and reasoning.
-
-${JSON.stringify(results, null, 2)}
-
-For each evaluation, score 1-3:
-- 1: Major issues (wrong tools, broken logic, incorrect output)
-- 2: Minor issues (inefficient but correct)
-- 3: Excellent (efficient trajectory, correct output)
-
-Respond as JSON array: [{"id": "...", "score": N, "reasoning": "..."}]
-`
-
-const response = await model.generateContent(prompt)
-console.log(response.response.text())
-```
-
-### Medium Context Models (Claude 200k)
-
-Use full trajectory for most runs:
-
-```typescript
-import Anthropic from '@anthropic-ai/sdk'
-
-const client = new Anthropic()
-const markdown = await Bun.file('results.md').text()
-
-const response = await client.messages.create({
-  model: 'claude-sonnet-4-20250514',
-  max_tokens: 4096,
-  messages: [{
-    role: 'user',
-    content: `Evaluate these agent trajectories:\n\n${markdown}\n\nScore each 1-3 and explain.`
-  }]
-})
-
-console.log(response.content[0].text)
-```
-
-## Braintrust Integration
-
-Upload results programmatically:
-
-```typescript
-import { initLogger } from 'braintrust'
-
-const logger = initLogger({
-  projectName: 'agent-eval',
-  apiKey: process.env.BRAINTRUST_API_KEY,
-})
-
-const results = parseResults(await Bun.file('results.jsonl').text())
-
-for (const result of results) {
-  logger.log({
-    input: result.input,
-    output: result.output,
-    expected: result.expected,
-    scores: {
-      passed: result.status === 'passed' ? 1 : 0,
-      duration_ms: result.duration,
-    },
-    metadata: {
-      ...result.metadata,
-      toolCalls: result.toolCalls,
-    },
-  })
-}
-
-await logger.flush()
-```
-
-## CI Integration
-
-### GitHub Actions
-
-```yaml
-name: Agent Eval
-on:
-  schedule:
-    - cron: '0 0 * * 0'  # Weekly
-
-jobs:
-  eval:
-    runs-on: ubuntu-latest
-    steps:
-      - uses: actions/checkout@v4
-      - uses: oven-sh/setup-bun@v2
-
-      - name: Install ACP adapter
-        run: npm install -g @zed-industries/claude-code-acp
-
-      - name: Install dependencies
-        run: bun add @plaited/acp-harness
-
-      - name: Run harness
-        env:
-          ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
-        run: |
-          bunx @plaited/acp-harness prompts.jsonl \
-            --format judge \
-            --progress \
-            -o eval-results
-
-      - name: Upload results
-        uses: actions/upload-artifact@v4
-        with:
-          name: eval-results
-          path: |
-            eval-results.md
-            eval-results.full.jsonl
-```
-
-## Output Aggregation
-
-Combine multiple runs:
-
-```bash
-# Append mode during runs
-bunx @plaited/acp-harness prompts-1.jsonl --append -o combined.jsonl
-bunx @plaited/acp-harness prompts-2.jsonl --append -o combined.jsonl
-
-# Merge separate files
-cat run1.jsonl run2.jsonl run3.jsonl > combined.jsonl
-
-# Dedupe by ID (keep latest) - use TypeScript pattern above
-```