@plaited/acp-harness 0.2.6 → 0.3.1

package/LICENSE

@@ -1,6 +1,6 @@
 ISC License
 
-Copyright (c) 2025 Plaited
+Copyright (c) 2026 Plaited Labs
 
 Permission to use, copy, modify, and/or distribute this software for any
 purpose with or without fee is hereby granted, provided that the above

package/README.md

@@ -10,37 +10,63 @@ CLI tool for capturing agent trajectories from ACP-compatible agents. Execute pr
 
 ```bash
 # Run without installing
-bunx @plaited/acp-harness prompts.jsonl -o results.jsonl
+bunx @plaited/acp-harness capture prompts.jsonl bunx claude-code-acp -o results.jsonl
 
 # Or install globally
 bun add -g @plaited/acp-harness
-acp-harness prompts.jsonl -o results.jsonl
+acp-harness capture prompts.jsonl bunx claude-code-acp -o results.jsonl
 ```
 
 **Prerequisite:** Install an ACP adapter and set your API key:
 
 ```bash
-npm install -g @zed-industries/claude-code-acp
+npm install -g @anthropic-ai/claude-code-acp
 export ANTHROPIC_API_KEY=sk-...
 ```
 
-## Usage
+## Commands
+
+| Command | Purpose |
+|---------|---------|
+| `capture` | Trajectory capture (full JSONL) |
+| `trials` | Multi-run with pass@k metrics |
+| `summarize` | Derive compact views from results |
+| `calibrate` | Sample failures for review |
+| `validate-refs` | Check reference solutions |
+| `balance` | Analyze test set coverage |
+| `schemas` | Export JSON schemas |
+
+### capture
+
+Capture full trajectories from an ACP agent:
 
 ```bash
-acp-harness <prompts.jsonl> [options]
+acp-harness capture <prompts.jsonl> <command> [args...] [options]
 
 Options:
-  --cmd, --command  ACP agent command (default: "claude-code-acp")
   -o, --output      Output file (default: stdout)
   -c, --cwd         Working directory for agent
   -t, --timeout     Request timeout in ms (default: 60000)
-  -f, --format      Output format: summary, judge (default: summary)
+  -g, --grader      Path to grader (.ts/.js module or executable script)
   --progress        Show progress to stderr
   --append          Append to output file
   --mcp-server      MCP server config JSON (repeatable)
   -h, --help        Show help
 ```
 
+### trials
+
+Run multiple trials per prompt for pass@k analysis:
+
+```bash
+acp-harness trials <prompts.jsonl> <command> [args...] [options]
+
+Options:
+  -k                Number of trials per prompt (default: 3)
+  -g, --grader      Path to grader (computes pass@k/pass^k metrics)
+  ...               (same as capture)
+```
+
 ## Input Format
 
 ```jsonl
@@ -48,16 +74,94 @@ Options:
 {"id":"test-002","input":"Fix the TypeScript error","metadata":{"category":"bugfix"}}
 ```
 
-## Output
+## Output Format
+
+The harness outputs full trajectory JSONL (`CaptureResult` schema):
+
+```jsonl
+{
+  "id": "test-001",
+  "input": "Create a primary button",
+  "output": "Here's a button component...",
+  "expected": "should contain <button>",
+  "trajectory": [...],
+  "metadata": {"category": "ui", "agent": "bunx claude-code-acp"},
+  "timing": {"start": 1234567890, "end": 1234567900},
+  "toolErrors": false,
+  "score": {"pass": true, "score": 1.0, "reasoning": "Contains expected"}
+}
+```
+
+Key fields:
+- `toolErrors`: Boolean indicating if any tool calls failed
+- `score`: Grader result (only if `--grader` provided)
+- `trajectory`: Full execution trace (thoughts, messages, tool calls, plans)
+
+## Graders
+
+Graders score agent outputs. The harness supports two types:
+
+### TypeScript/JavaScript Graders
+
+Export a `grade` function:
+
+```typescript
+import type { Grader } from '@plaited/acp-harness/schemas'
+
+export const grade: Grader = async ({ input, output, expected, trajectory }) => {
+  const pass = output.toLowerCase().includes(expected?.toLowerCase() ?? '')
+  return {
+    pass,
+    score: pass ? 1.0 : 0.0,
+    reasoning: pass ? 'Contains expected answer' : 'Missing expected answer'
+  }
+}
+```
+
+```bash
+acp-harness capture prompts.jsonl bunx claude-code-acp --grader ./grader.ts
+```
+
+### Polyglot Graders (Python, etc.)
+
+Any executable script using stdin/stdout JSON protocol:
+
+```python
+#!/usr/bin/env python3
+import json
+import sys
+
+data = json.load(sys.stdin)
+output = data["output"].lower()
+expected = (data.get("expected") or "").lower()
+
+pass_result = expected in output if expected else True
+print(json.dumps({
+    "pass": pass_result,
+    "score": 1.0 if pass_result else 0.0,
+    "reasoning": "Contains expected" if pass_result else "Missing expected"
+}))
+```
+
+```bash
+chmod +x grader.py
+acp-harness capture prompts.jsonl bunx claude-code-acp --grader ./grader.py
+```
 
-The harness captures trajectories and outputs structured JSONL. **You provide the scoring logic.**
+**Protocol:**
+- Input (stdin): `{"input": "...", "output": "...", "expected": "...", "trajectory": [...]}`
+- Output (stdout): `{"pass": true, "score": 1.0, "reasoning": "..."}`
+
+## Downstream Integration
 
 ```bash
-# Capture trajectories
-acp-harness prompts.jsonl -o results.jsonl
+# Filter failures
+cat results.jsonl | jq 'select(.score.pass == false)'
+
+# Extract tool usage patterns
+cat results.jsonl | jq '.trajectory[] | select(.type == "tool_call") | .name'
 
-# Score with your tools
-cat results.jsonl | jq 'select(.status == "failed")'
+# Use with your scoring pipeline
 cat results.jsonl | your-scoring-script.ts
 ```
 
@@ -69,10 +173,10 @@ This package includes an **acp-harness skill** for AI coding agents with complet
 - Output format schemas
 - Integration patterns (Braintrust, jq, custom scorers)
 
-**Install via Claude Code:**
+**Other AI coding agents:**
 
 ```bash
-/plugin marketplace add plaited/marketplace
+curl -fsSL https://raw.githubusercontent.com/plaited/marketplace/main/install.sh | bash -s -- --agent <agent-name> --plugin development-skills
 ```
 
 ## Development
@@ -86,7 +190,7 @@ bun test             # Run unit tests
 ## Requirements
 
 - **Runtime:** Bun >= 1.2.9
-- **ACP Adapter:** `@zed-industries/claude-code-acp` or compatible
+- **ACP Adapter:** `@anthropic-ai/claude-code-acp` or compatible
 - **API Key:** `ANTHROPIC_API_KEY` environment variable
 
 ## License