harness-evolver 0.1.0

package/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Raphael Valdetaro Christi Cordeiro
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

package/README.md

@@ -0,0 +1,252 @@
+# Harness Evolver
+
+End-to-end optimization of LLM agent harnesses, inspired by [Meta-Harness](https://yoonholee.com/meta-harness/) (Lee et al., 2026).
+
+**The harness is the 80% factor.** Changing just the scaffolding around a fixed LLM can produce a [6x performance gap](https://arxiv.org/abs/2603.28052) on the same benchmark. Harness Evolver automates the search for better harnesses using an autonomous propose-evaluate-iterate loop with full execution traces as feedback.
+
+## Why
+
+Manual harness engineering is slow and doesn't scale. Existing optimizers work in prompt-space (OPRO, TextGrad, GEPA) or use compressed summaries. Meta-Harness showed that **code-space search with full diagnostic context** (10M+ tokens of traces) outperforms all of them by 10+ points.
+
+Harness Evolver brings that approach to any domain as a Claude Code plugin.
+
+## Install
+
+```bash
+# Via npx (recommended)
+npx harness-evolver@latest
+
+# Or as a Claude Code plugin
+/plugin install harness-evolver
+```
+
+## Quick Start
+
+```bash
+# 1. Copy the example into a working directory
+cp -r ~/.harness-evolver/examples/classifier ./my-classifier
+cd my-classifier
+
+# 2. Initialize (validates harness, evaluates baseline)
+/harness-evolve-init --harness harness.py --eval eval.py --tasks tasks/
+
+# 3. Run the evolution loop
+/harness-evolve --iterations 5
+
+# 4. Check progress anytime
+/harness-evolve-status
+```
+
+The classifier example runs in mock mode (no API key needed) and demonstrates the full loop in under 2 minutes.
+
+## How It Works
+
+```
+                    ┌─────────────────────────────┐
+                    │     /harness-evolve          │
+                    │     (orchestrator skill)     │
+                    └──────────┬──────────────────┘
+                               │
+              ┌────────────────┼────────────────┐
+              ▼                ▼                ▼
+        ┌──────────┐   ┌────────────┐   ┌──────────┐
+        │ PROPOSE  │   │  EVALUATE  │   │  UPDATE   │
+        │ proposer │   │ evaluate.py│   │ state.py  │
+        │ agent    │   │ + eval.py  │   │           │
+        └──────────┘   └────────────┘   └──────────┘
+              │                │                │
+              ▼                ▼                ▼
+        harnesses/       traces/         summary.json
+        v{N}/            per-task        STATE.md
+        harness.py       stdout/stderr   PROPOSER_HISTORY.md
+        proposal.md      timing.json
+        scores.json
+```
+
+1. **Propose** — A proposer agent (Claude Code subagent) reads all prior candidates' code, execution traces, and scores. It diagnoses failure modes via counterfactual analysis and writes a new harness.
+2. **Evaluate** — The harness runs against every task. Traces are captured per-task (input, output, stdout, stderr, timing). The user's eval script scores the results.
+3. **Update** — State files are updated with the new score, parent lineage, and regression detection.
+4. **Repeat** — The loop continues until N iterations, stagnation (3 rounds without >1% improvement), or a target score is reached.
+
+## The Harness Contract
+
+A harness is **any executable** that accepts:
+
+```bash
+python3 harness.py --input task.json --output result.json [--traces-dir DIR] [--config config.json]
+```
+
+- `--input`: JSON with `{id, input, metadata}` (never sees expected answers)
+- `--output`: JSON with `{id, output}`
+- `--traces-dir`: optional directory for the harness to write rich traces
+- `--config`: optional JSON with evolvable parameters (model, temperature, etc.)
+
+The eval script is also any executable:
+
+```bash
+python3 eval.py --results-dir results/ --tasks-dir tasks/ --scores scores.json
+```
+
+This means Harness Evolver works with **any language, any framework, any domain**.
+
+## Project Structure
+
+```
+.harness-evolver/                     # Created in your project by /harness-evolve-init
+├── config.json                       # Project config (harness cmd, eval cmd, evolution params)
+├── summary.json                      # Source of truth (versions, scores, parents)
+├── STATE.md                          # Human-readable status (generated)
+├── PROPOSER_HISTORY.md               # Log of all proposals and outcomes
+├── baseline/                         # Original harness (read-only reference)
+│   ├── harness.py
+│   └── config.json
+├── eval/
+│   ├── eval.py                       # Scoring script
+│   └── tasks/                        # Test cases (JSON files)
+└── harnesses/
+    └── v001/
+        ├── harness.py                # Candidate code
+        ├── config.json               # Evolvable parameters
+        ├── proposal.md               # Proposer's reasoning
+        ├── scores.json               # Evaluation results
+        └── traces/                   # Full execution traces
+            ├── stdout.log
+            ├── stderr.log
+            ├── timing.json
+            └── task_001/
+                ├── input.json        # What the harness received
+                └── output.json       # What the harness returned
+```
+
+## Plugin Architecture
+
+Three-layer design inspired by [GSD](https://github.com/gsd-build/get-shit-done):
+
+```
+Layer 1: Skills + Agents (markdown)     → AI orchestration
+Layer 2: Tools (Python stdlib-only)     → Deterministic operations
+Layer 3: Installer (Node.js)            → Distribution via npx
+```
+
+| Component | Files | Purpose |
+|---|---|---|
+| **Skills** | `skills/harness-evolve-init/`, `skills/harness-evolve/`, `skills/harness-evolve-status/` | Slash commands that orchestrate the loop |
+| **Agent** | `agents/harness-evolver-proposer.md` | The proposer — 4-phase workflow (orient, diagnose, propose, document) with 6 rules |
+| **Tools** | `tools/evaluate.py`, `tools/state.py`, `tools/init.py`, `tools/detect_stack.py`, `tools/trace_logger.py` | CLI tools called via subprocess — zero LLM tokens spent on deterministic work |
+| **Installer** | `bin/install.js`, `package.json` | Copies skills/agents/tools to the right locations |
+| **Example** | `examples/classifier/` | 10-task medical classifier with mock mode |
+
+## Integrations
+
+### LangSmith (optional)
+
+If `LANGSMITH_API_KEY` is set, the plugin automatically:
+- Enables `LANGCHAIN_TRACING_V2` for auto-tracing of LangChain/LangGraph harnesses
+- Detects [langsmith-cli](https://github.com/gigaverse-app/langsmith-cli) for the proposer to query traces directly
+
+```bash
+# Setup
+export LANGSMITH_API_KEY=lsv2_...
+uv tool install langsmith-cli && langsmith-cli auth login
+
+# The proposer can then do:
+langsmith-cli --json runs list --project harness-evolver-v003 --failed --fields id,name,error
+langsmith-cli --json runs stats --project harness-evolver-v003
+```
+
+No custom API client — the proposer uses `langsmith-cli` like it uses `grep` and `diff`.
+
+### Context7 (optional)
+
+The plugin detects the harness's technology stack via AST analysis (17 libraries supported) and instructs the proposer to consult current documentation before proposing API changes.
+
+```bash
+# Setup
+claude mcp add context7 -- npx -y @upstash/context7-mcp@latest
+
+# The proposer automatically:
+# 1. Reads config.json → stack.detected (e.g., LangChain, ChromaDB)
+# 2. Queries Context7 for current docs before writing code
+# 3. Annotates proposal.md with "API verified via Context7"
+```
+
+Without Context7, the proposer uses model knowledge and annotates "API not verified against current docs."
+
+### LangChain Docs MCP (optional)
+
+```bash
+claude mcp add docs-langchain --transport http https://docs.langchain.com/mcp
+```
+
+Complements Context7 with LangChain/LangGraph/LangSmith-specific documentation search.
+
+## The Proposer
+
+The proposer agent is the core of the system. It follows a 4-phase workflow derived from the Meta-Harness paper:
+
+| Phase | Context % | What it does |
+|---|---|---|
+| **Orient** | ~6% | Read `summary.json` and `PROPOSER_HISTORY.md`. Decide which 2-3 versions to investigate. |
+| **Diagnose** | ~80% | Deep trace analysis on selected versions. grep for errors, diff between good/bad versions, counterfactual diagnosis. |
+| **Propose** | ~10% | Write new `harness.py` + `config.json`. Prefer additive changes after regressions. |
+| **Document** | ~4% | Write `proposal.md` with evidence. Append to `PROPOSER_HISTORY.md`. |
+
+**6 rules:**
+1. Every change motivated by evidence (cite task ID, trace line, or score delta)
+2. After regression, prefer additive changes
+3. Don't repeat past mistakes (read PROPOSER_HISTORY.md)
+4. One hypothesis at a time when possible
+5. Maintain the CLI interface
+6. Prefer readable harnesses over defensive ones
+
+## Supported Libraries (Stack Detection)
+
+The AST-based stack detector recognizes 17 libraries:
+
+| Category | Libraries |
+|---|---|
+| **AI Frameworks** | LangChain, LangGraph, LlamaIndex, OpenAI, Anthropic, DSPy, CrewAI, AutoGen |
+| **Vector Stores** | ChromaDB, Pinecone, Qdrant, Weaviate |
+| **Web** | FastAPI, Flask, Pydantic |
+| **Data** | Pandas, NumPy |
+
+## Development
+
+```bash
+# Run all tests (41 tests, stdlib-only, no pip install needed)
+python3 -m unittest discover -s tests -v
+
+# Test the example manually
+cd examples/classifier
+python3 harness.py --input tasks/task_001.json --output /tmp/result.json --config config.json
+cat /tmp/result.json
+
+# Run the installer locally
+node bin/install.js
+```
+
+## Comparison with Related Work
+
+| | Meta-Harness (paper) | A-Evolve | ECC /evolve | **Harness Evolver** |
+|---|---|---|---|---|
+| **Format** | Paper artifact | Framework (Docker) | Plugin (passive) | **Plugin (active)** |
+| **Search space** | Code-space | Code-space | Prompt-space | **Code-space** |
+| **Context/iter** | 10M tokens | Variable | N/A | **Full filesystem** |
+| **Domain** | TerminalBench-2 | Coding benchmarks | Dev workflow | **Any domain** |
+| **Install** | Manual Python | Docker CLI | `/plugin install` | **`npx` or `/plugin install`** |
+| **LangSmith** | No | No | No | **Yes (langsmith-cli)** |
+| **Context7** | No | No | No | **Yes (MCP)** |
+
+## References
+
+- [Meta-Harness: End-to-End Optimization of Model Harnesses](https://arxiv.org/abs/2603.28052) — Lee et al., 2026
+- [GSD (Get Shit Done)](https://github.com/gsd-build/get-shit-done) — CLI architecture inspiration
+- [LangSmith CLI](https://github.com/gigaverse-app/langsmith-cli) — Trace analysis for the proposer
+- [Context7](https://github.com/upstash/context7) — Documentation lookup via MCP
+- [Design Spec](docs/specs/2026-03-31-harness-evolver-design.md)
+- [LangSmith Integration Spec](docs/specs/2026-03-31-langsmith-integration.md)
+- [Context7 Integration Spec](docs/specs/2026-03-31-context7-integration.md)
+
+## License
+
+MIT

package/agents/harness-evolver-proposer.md

@@ -0,0 +1,163 @@
+---
+name: harness-evolver-proposer
+description: |
+  Use this agent when the harness-evolve skill needs to propose a new harness candidate.
+  This agent navigates the .harness-evolver/ filesystem to diagnose failures in prior
+  candidates and propose an improved harness. It is the core of the Meta-Harness optimization loop.
+model: opus
+---
+
+# Harness Evolver — Proposer Agent
+
+You are the proposer in a Meta-Harness optimization loop. Your job is to analyze all prior harness candidates — their code, execution traces, and scores — and propose a new harness that improves on them.
+
+## Context
+
+You are working inside a `.harness-evolver/` directory with this structure:
+
+```
+.harness-evolver/
+├── summary.json              # Panorama: all versions, scores, parents
+├── PROPOSER_HISTORY.md       # Your prior decisions and their outcomes
+├── config.json               # Project config (harness command, eval command, etc.)
+├── baseline/
+│   ├── harness.py            # Original harness (read-only reference)
+│   └── config.json           # Original config
+├── eval/
+│   ├── eval.py               # Scoring script (DO NOT MODIFY)
+│   └── tasks/                # Test cases (DO NOT MODIFY)
+└── harnesses/
+    └── v001/
+        ├── harness.py        # Candidate code
+        ├── config.json       # Candidate params
+        ├── proposal.md       # Why this version exists
+        ├── scores.json       # How it scored
+        └── traces/
+            ├── stdout.log    # Raw stdout from harness runs
+            ├── stderr.log    # Raw stderr
+            ├── timing.json   # Per-task timing
+            └── task_001/
+                ├── input.json   # What the harness received
+                ├── output.json  # What the harness returned
+                └── extra/       # Optional traces from harness
+```
+
+## Your Workflow
+
+### Phase 1: ORIENT (read summary, identify focus)
+
+1. Read `summary.json` to see all versions, scores, and parent lineage.
+2. Read `PROPOSER_HISTORY.md` to see what you've tried before and what worked or failed.
+3. Decide which 2-3 versions to investigate deeply:
+   - (a) The current best candidate
+   - (b) The most recent regression (if any)
+   - (c) A version with a different failure mode
+
+### Phase 2: DIAGNOSE (deep trace analysis)
+
+Investigate the selected versions. Use standard tools:
+- `cat .harness-evolver/harnesses/v{N}/scores.json` — see per-task results
+- `cat .harness-evolver/harnesses/v{N}/traces/task_XXX/output.json` — see what went wrong
+- `cat .harness-evolver/harnesses/v{N}/traces/stderr.log` — look for errors
+- `diff .harness-evolver/harnesses/v{A}/harness.py .harness-evolver/harnesses/v{B}/harness.py` — compare
+- `grep -r "error\|Error\|FAIL\|exception" .harness-evolver/harnesses/v{N}/traces/`
+
+Ask yourself:
+- Which tasks fail? Is there a pattern?
+- What changed between a version that passed and one that failed?
+- Is this a code bug, a prompt issue, a retrieval problem, or a parameter problem?
+
+**Do NOT read traces of all versions.** Focus on 2-3. Use summary.json to filter.
+
+### Phase 3: PROPOSE (write new harness)
+
+Based on your diagnosis, create a new version directory and write:
+
+1. `harnesses/v{NEXT}/harness.py` — the new harness code
+2. `harnesses/v{NEXT}/config.json` — parameters (copy from parent, modify if needed)
+3. `harnesses/v{NEXT}/proposal.md` — your reasoning (MUST include "Based on v{PARENT}")
+
+**The harness MUST maintain this CLI interface:**
+```
+python3 harness.py --input INPUT.json --output OUTPUT.json [--traces-dir DIR] [--config CONFIG.json]
+```
+
+### Phase 4: DOCUMENT
+
+Write a clear `proposal.md` that includes:
+- `Based on v{PARENT}` on the first line
+- What failure modes you identified
+- What specific changes you made and why
+- What you expect to improve
+
+Append a summary to `PROPOSER_HISTORY.md`.
+
+## Rules
+
+1. **Every change motivated by evidence.** Cite the task ID, trace line, or score delta that justifies the change. Never change code "to see what happens."
+
+2. **After a regression, prefer additive changes.** If the last version regressed, make smaller, safer modifications. Don't combine multiple changes.
+
+3. **Don't repeat past mistakes.** Read PROPOSER_HISTORY.md. If an approach already failed (e.g., "changed prompt template, broke JSON parsing"), don't try a similar approach without strong justification.
+
+4. **One hypothesis at a time when possible.** Changing A+B+C simultaneously makes it impossible to diagnose which helped or hurt. If you must make multiple changes, document each clearly.
+
+5. **Maintain the interface.** The harness must accept --input, --output, --traces-dir, --config. Breaking the interface breaks the entire loop.
+
+6. **Prefer readable harnesses over defensive ones.** If the harness has grown past 2x the baseline size without proportional score improvement, consider simplifying. Accumulated try/catch blocks, redundant fallbacks, and growing if-chains are a code smell in evolved harnesses.
+
+## Documentation Lookup (if Context7 available)
+
+- Read `config.json` field `stack.detected` to see which libraries the harness uses.
+- BEFORE writing code that uses a library from the detected stack,
+  use the `resolve-library-id` tool with the `context7_id` from the config, then
+  `get-library-docs` to fetch documentation relevant to your proposed change.
+- If Context7 is NOT available, proceed with model knowledge
+  but note in `proposal.md`: "API not verified against current docs."
+- Do NOT look up docs for every line of code — only when proposing
+  changes that involve specific APIs (new imports, new methods, new parameters).
+
+## What You Do NOT Do
+
+- Do NOT run the evaluation. The evolve skill handles that after you propose.
+- Do NOT modify anything in `eval/` — the eval set and scoring are fixed.
+- Do NOT modify `baseline/` — it is your immutable reference.
+- Do NOT modify any prior version's files — history is immutable.
+- Do NOT create files outside of `harnesses/v{NEXT}/` and `PROPOSER_HISTORY.md`.
+
+## LangSmith Traces (when langsmith-cli is available)
+
+If LangSmith tracing is enabled (check `config.json` field `eval.langsmith.enabled`),
+each harness run is automatically traced to a LangSmith project named
+`{project_prefix}-v{NNN}`.
+
+Use `langsmith-cli` to query traces directly:
+
+```bash
+# Find failures in this version
+langsmith-cli --json runs list --project harness-evolver-v{N} --failed --fields id,name,error,inputs
+
+# Aggregate stats (error rate, latency p50/p95/p99)
+langsmith-cli --json runs stats --project harness-evolver-v{N}
+
+# Search for specific error patterns
+langsmith-cli --json runs list --grep "pattern" --grep-in error --project harness-evolver-v{N} --fields id,error
+
+# Compare two versions
+langsmith-cli --json runs stats --project harness-evolver-v{A}
+langsmith-cli --json runs stats --project harness-evolver-v{B}
+
+# Get full details of latest failure
+langsmith-cli --json runs get-latest --project harness-evolver-v{N} --failed
+```
+
+ALWAYS use `--json` as the first flag and `--fields` to limit output size.
+If `langsmith-cli` is not available, fall back to local traces in `traces/` as usual.
+
+## Output
+
+When done, report what you created:
+- Version number (e.g., "v003")
+- Parent version
+- 1-sentence summary of the change
+- Expected impact on score

package/bin/install.js

@@ -0,0 +1,125 @@
+#!/usr/bin/env node
+/**
+ * Harness Evolver installer.
+ * Detects Claude Code, copies skills/agents/tools to the right locations.
+ *
+ * Usage: npx harness-evolver@latest
+ */
+
+const fs = require("fs");
+const path = require("path");
+const { execSync } = require("child_process");
+
+const PLUGIN_ROOT = path.resolve(__dirname, "..");
+const HOME = process.env.HOME || process.env.USERPROFILE;
+
+const CLAUDE_DIR = path.join(HOME, ".claude");
+const COMMANDS_DIR = path.join(CLAUDE_DIR, "commands", "harness-evolver");
+const AGENTS_DIR = path.join(CLAUDE_DIR, "agents");
+const TOOLS_DIR = path.join(HOME, ".harness-evolver", "tools");
+const EXAMPLES_DIR = path.join(HOME, ".harness-evolver", "examples");
+
+function log(msg) {
+  console.log(`  ${msg}`);
+}
+
+function copyDir(src, dest) {
+  fs.mkdirSync(dest, { recursive: true });
+  for (const entry of fs.readdirSync(src, { withFileTypes: true })) {
+    const srcPath = path.join(src, entry.name);
+    const destPath = path.join(dest, entry.name);
+    if (entry.isDirectory()) {
+      copyDir(srcPath, destPath);
+    } else {
+      fs.copyFileSync(srcPath, destPath);
+    }
+  }
+}
+
+function copyFile(src, dest) {
+  fs.mkdirSync(path.dirname(dest), { recursive: true });
+  fs.copyFileSync(src, dest);
+}
+
+function checkPython() {
+  try {
+    execSync("python3 --version", { stdio: "pipe" });
+    return true;
+  } catch {
+    return false;
+  }
+}
+
+function main() {
+  console.log("\n  Harness Evolver v0.1.0\n");
+
+  if (!checkPython()) {
+    console.error("  ERROR: python3 not found in PATH. Install Python 3.8+ first.");
+    process.exit(1);
+  }
+  log("\u2713 python3 found");
+
+  if (!fs.existsSync(CLAUDE_DIR)) {
+    console.error(`  ERROR: Claude Code directory not found at ${CLAUDE_DIR}`);
+    console.error("  Install Claude Code first: https://claude.ai/code");
+    process.exit(1);
+  }
+  log("\u2713 Claude Code detected");
+
+  // Copy skills
+  const skillsSource = path.join(PLUGIN_ROOT, "skills");
+  if (fs.existsSync(skillsSource)) {
+    for (const skill of fs.readdirSync(skillsSource, { withFileTypes: true })) {
+      if (skill.isDirectory()) {
+        const src = path.join(skillsSource, skill.name);
+        const dest = path.join(COMMANDS_DIR, skill.name);
+        copyDir(src, dest);
+        log(`  skill: ${skill.name}`);
+      }
+    }
+  }
+
+  // Copy agents
+  const agentsSource = path.join(PLUGIN_ROOT, "agents");
+  if (fs.existsSync(agentsSource)) {
+    fs.mkdirSync(AGENTS_DIR, { recursive: true });
+    for (const agent of fs.readdirSync(agentsSource)) {
+      copyFile(
+        path.join(agentsSource, agent),
+        path.join(AGENTS_DIR, agent)
+      );
+      log(`  agent: ${agent}`);
+    }
+  }
+
+  // Copy tools
+  const toolsSource = path.join(PLUGIN_ROOT, "tools");
+  if (fs.existsSync(toolsSource)) {
+    fs.mkdirSync(TOOLS_DIR, { recursive: true });
+    for (const tool of fs.readdirSync(toolsSource)) {
+      if (tool.endsWith(".py")) {
+        copyFile(
+          path.join(toolsSource, tool),
+          path.join(TOOLS_DIR, tool)
+        );
+        log(`  tool: ${tool}`);
+      }
+    }
+  }
+
+  // Copy examples
+  const examplesSource = path.join(PLUGIN_ROOT, "examples");
+  if (fs.existsSync(examplesSource)) {
+    copyDir(examplesSource, EXAMPLES_DIR);
+    log("  examples: classifier");
+  }
+
+  console.log("\n  \u2713 Installed successfully!\n");
+  console.log("  Next steps:");
+  console.log("    1. Copy an example:  cp -r ~/.harness-evolver/examples/classifier ./my-project");
+  console.log("    2. cd my-project");
+  console.log("    3. /harness-evolve-init --harness harness.py --eval eval.py --tasks tasks/");
+  console.log("    4. /harness-evolve --iterations 5\n");
+}
+
+main();

package/examples/classifier/README.md

@@ -0,0 +1,25 @@
+# Classifier Example
+
+Medical symptom classifier — deliberately naive, designed to be improved by the evolver.
+
+## Quick Start (Mock Mode — No API Key)
+
+```bash
+/harness-evolve-init --harness harness.py --eval eval.py --tasks tasks/
+/harness-evolve --iterations 5
+```
+
+## With LLM
+
+Edit `config.json`:
+```json
+{
+  "mock": false,
+  "api_key": "sk-ant-...",
+  "model": "claude-haiku-4-5-20251001"
+}
+```
+
+## Categories
+
+respiratory, cardiac, gastrointestinal, neurological, musculoskeletal, dermatological

package/examples/classifier/config.json

@@ -0,0 +1,3 @@
+{
+  "mock": true
+}

package/examples/classifier/eval.py

@@ -0,0 +1,58 @@
+#!/usr/bin/env python3
+"""Exact match accuracy scorer for the classifier example."""
+
+import argparse
+import json
+import os
+
+
+def main():
+    parser = argparse.ArgumentParser()
+    parser.add_argument("--results-dir", required=True)
+    parser.add_argument("--tasks-dir", required=True)
+    parser.add_argument("--scores", required=True)
+    args = parser.parse_args()
+
+    correct = 0
+    total = 0
+    per_task = {}
+
+    for fname in sorted(os.listdir(args.tasks_dir)):
+        if not fname.endswith(".json"):
+            continue
+        task_path = os.path.join(args.tasks_dir, fname)
+        task = json.load(open(task_path))
+        task_id = task["id"]
+
+        result_path = os.path.join(args.results_dir, fname)
+        if not os.path.exists(result_path):
+            per_task[task_id] = {"score": 0.0, "error": "no output file"}
+            total += 1
+            continue
+
+        result = json.load(open(result_path))
+        expected = task["expected"].lower().strip()
+        actual = result.get("output", "").lower().strip()
+        match = actual == expected
+
+        per_task[task_id] = {
+            "score": 1.0 if match else 0.0,
+            "expected": expected,
+            "actual": actual,
+        }
+        correct += int(match)
+        total += 1
+
+    accuracy = correct / total if total > 0 else 0.0
+    scores = {
+        "combined_score": accuracy,
+        "accuracy": accuracy,
+        "total_tasks": total,
+        "correct": correct,
+        "per_task": per_task,
+    }
+    json.dump(scores, open(args.scores, "w"), indent=2)
+
+
+if __name__ == "__main__":
+    main()