harness-evolver 2.9.0 → 3.0.0

package/skills/setup/SKILL.md

@@ -0,0 +1,182 @@
+---
+name: evolver:setup
+description: "Use when the user wants to set up the evolver in their project, optimize an LLM agent, improve agent performance, or mentions evolver for the first time in a project without .evolver.json."
+argument-hint: "[directory]"
+allowed-tools: [Read, Write, Edit, Bash, Glob, Grep, Agent, AskUserQuestion]
+---
+
+# /evolver:setup
+
+Set up the Harness Evolver v3 in a project. Explores the codebase, configures LangSmith, runs baseline evaluation.
+
+## Prerequisites
+
+- `LANGSMITH_API_KEY` must be set. If not: "Set your LangSmith API key: `export LANGSMITH_API_KEY=lsv2_pt_...`"
+- Python 3.10+ with `langsmith` and `openevals` packages. If missing:
+
+```bash
+pip install langsmith openevals 2>/dev/null || uv pip install langsmith openevals
+```
+
+## Resolve Tool Path
+
+```bash
+TOOLS=$([ -d ".evolver/tools" ] && echo ".evolver/tools" || echo "$HOME/.evolver/tools")
+```
+
+## Phase 1: Explore Project (automatic)
+
+```bash
+find . -maxdepth 3 -type f -name "*.py" | head -30
+python3 $TOOLS/detect_stack.py .
+```
+
+Look for:
+- Entry points: files with `if __name__`, or named `main.py`, `app.py`, `agent.py`, `graph.py`, `pipeline.py`
+- Framework: LangGraph, CrewAI, OpenAI SDK, Anthropic SDK, etc.
+- Existing LangSmith config: `LANGCHAIN_PROJECT` / `LANGSMITH_PROJECT` in env or `.env`
+- Existing test data: JSON files with inputs, CSV files, etc.
+- Dependencies: `requirements.txt`, `pyproject.toml`
+
+Identify the **run command** — how to execute the agent. Examples:
+- `python main.py` (if it accepts `--input` flag)
+- `python -c "from agent import run; import json,sys; print(json.dumps(run(json.load(open(sys.argv[1])))))"`
+
+## Phase 2: Confirm Detection (interactive)
+
+Use AskUserQuestion:
+
+```json
+{
+  "questions": [{
+    "question": "Here's what I detected. Does this look right?\n\nEntry point: {path}\nFramework: {framework}\nRun command: {command}\nLangSmith: {status}",
+    "header": "Confirm",
+    "multiSelect": false,
+    "options": [
+      {"label": "Looks good, proceed", "description": "Continue with detected configuration"},
+      {"label": "Let me adjust", "description": "I'll provide correct paths and commands"},
+      {"label": "Wrong directory", "description": "I need to cd somewhere else first"}
+    ]
+  }]
+}
+```
+
+## Phase 3: What to Optimize (interactive)
+
+Use AskUserQuestion:
+
+```json
+{
+  "questions": [{
+    "question": "What do you want to optimize?",
+    "header": "Goals",
+    "multiSelect": true,
+    "options": [
+      {"label": "Accuracy", "description": "Correctness of outputs — LLM-as-judge evaluator"},
+      {"label": "Latency", "description": "Response time — track and minimize"},
+      {"label": "Token efficiency", "description": "Fewer tokens for same quality"},
+      {"label": "Error handling", "description": "Reduce failures, timeouts, crashes"}
+    ]
+  }]
+}
+```
+
+Map selections to evaluator configuration for setup.py.
+
+## Phase 4: Test Data Source (interactive)
+
+Use AskUserQuestion with **preview**:
+
+```json
+{
+  "questions": [{
+    "question": "Where should test inputs come from?",
+    "header": "Test data",
+    "multiSelect": false,
+    "options": [
+      {
+        "label": "Import from LangSmith",
+        "description": "Use real production traces as test inputs",
+        "preview": "## Import from LangSmith\n\nFetches up to 100 recent traces from your production project.\nPrioritizes traces with negative feedback.\nCreates a LangSmith Dataset with real user inputs.\n\nRequires: an existing LangSmith project with traces."
+      },
+      {
+        "label": "Generate from code",
+        "description": "AI generates test inputs by analyzing your code",
+        "preview": "## Generate from Code\n\nThe testgen agent reads your source code and generates\n30 diverse test inputs:\n- 40% standard cases\n- 20% edge cases\n- 20% cross-domain\n- 20% adversarial\n\nOutputs are scored by LLM-as-judge."
+      },
+      {
+        "label": "I have test data",
+        "description": "Point to an existing file with test inputs",
+        "preview": "## Provide Test Data\n\nSupported formats:\n- JSON array of inputs\n- JSON with {\"inputs\": {...}} objects\n- CSV with input columns\n\nExample:\n```json\n[\n  {\"input\": \"What is Python?\"},\n  {\"input\": \"Explain quantum computing\"}\n]\n```"
+      }
+    ]
+  }]
+}
+```
+
+If "Import from LangSmith": discover projects and ask which one (same as v2 Phase 1.9).
+If "I have test data": ask for file path.
+
+## Phase 5: Run Setup
+
+Build the setup.py command based on all gathered information:
+
+```bash
+python3 $TOOLS/setup.py \
+    --project-name "{project_name}" \
+    --entry-point "{run_command}" \
+    --framework "{framework}" \
+    --goals "{goals_csv}" \
+    ${DATASET_FROM_FILE:+--dataset-from-file "$DATASET_FROM_FILE"} \
+    ${DATASET_FROM_LANGSMITH:+--dataset-from-langsmith "$DATASET_FROM_LANGSMITH"} \
+    ${PRODUCTION_PROJECT:+--production-project "$PRODUCTION_PROJECT"}
+```
+
+If "Generate from code" was selected AND no test data file exists, first spawn the testgen agent to generate inputs, then pass the generated file to setup.py.
+
+## Phase 6: Generate Test Data (if needed)
+
+If testgen is needed, spawn it:
+
+```
+Agent(
+  subagent_type: "evolver-testgen",
+  description: "TestGen: generate test inputs",
+  prompt: |
+    <objective>
+    Generate 30 diverse test inputs for this project.
+    Write them as a JSON array to test_inputs.json.
+    </objective>
+
+    <files_to_read>
+    {all .py files discovered in Phase 1}
+    </files_to_read>
+
+    <output>
+    Create test_inputs.json with format:
+    [{"input": "..."}, {"input": "..."}, ...]
+    </output>
+)
+```
+
+Then pass `--dataset-from-file test_inputs.json` to setup.py.
+
+## Phase 7: Report
+
+```
+Setup complete!
+  Project: evolver-{name}
+  Dataset: {name}-eval-v1 ({N} examples)
+  Evaluators: {list}
+  Baseline score: {score}
+  Config: .evolver.json
+
+Next: run /evolver:evolve to start optimizing.
+```
+
+## Gotchas
+
+- If `.evolver.json` already exists, ask before overwriting.
+- If the agent needs a venv, the run command should activate it: `cd {dir} && .venv/bin/python main.py`
+- If LangSmith connection fails, check API key and network.
+- The setup installs `langsmith` and `openevals` if missing.

package/skills/status/SKILL.md

@@ -1,34 +1,36 @@
 ---
-name: harness-evolver:status
-description: "Use when the user asks about evolution progress, current scores, best harness version, how many iterations ran, or whether the loop is stagnating. Also use when the user says 'status', 'progress', or 'how is it going'."
+name: evolver:status
+description: "Use when the user asks about evolution progress, current scores, best version, how many iterations ran, or whether the loop is stagnating."
 allowed-tools: [Read, Bash]
 ---
 
-# /harness-evolve-status
+# /evolver:status
 
-Show evolution progress.
-
-## Resolve Tool Path
-
-```bash
-TOOLS=$([ -d ".harness-evolver/tools" ] && echo ".harness-evolver/tools" || echo "$HOME/.harness-evolver/tools")
-```
+Show current evolution progress.
 
 ## What To Do
 
-If `.harness-evolver/` does not exist, tell user to run `harness-evolver:init` first.
-
-Otherwise:
+Read `.evolver.json` and report:
 
 ```bash
-python3 $TOOLS/state.py show --base-dir .harness-evolver
+python3 -c "
+import json
+c = json.load(open('.evolver.json'))
+print(f'Project: {c[\"project\"]}')
+print(f'Dataset: {c[\"dataset\"]}')
+print(f'Framework: {c[\"framework\"]}')
+print(f'Evaluators: {c[\"evaluators\"]}')
+print(f'Iterations: {c[\"iterations\"]}')
+print(f'Best: {c[\"best_experiment\"]} (score: {c[\"best_score\"]:.3f})')
+print(f'Baseline: {c[\"history\"][0][\"score\"]:.3f}' if c['history'] else 'No baseline')
+print()
+print('History:')
+for h in c.get('history', []):
+    print(f'  {h[\"version\"]}: {h[\"score\"]:.3f}')
+"
 ```
 
-Then read and display `.harness-evolver/STATE.md` for the full history table.
-
-## If User Wants More Detail
+Detect stagnation: if last 3 scores are within 1% of each other, warn.
+Detect regression: if current best is lower than a previous best, warn.
 
-- Scores per task: `cat .harness-evolver/harnesses/{version}/scores.json`
-- What changed: `cat .harness-evolver/harnesses/{version}/proposal.md`
-- Compare two versions: `diff .harness-evolver/harnesses/{vA}/harness.py .harness-evolver/harnesses/{vB}/harness.py`
-- Full history: `cat .harness-evolver/PROPOSER_HISTORY.md`
+Print LangSmith URL for the best experiment if available.

package/tools/read_results.py

@@ -0,0 +1,240 @@
+#!/usr/bin/env python3
+"""Read LangSmith experiment results for Harness Evolver v3.
+
+Reads experiment results from LangSmith and formats them for agents
+(proposer, critic, architect). Handles comparison between candidates.
+
+Usage:
+    python3 read_results.py \
+        --experiments v001a,v001b,v001c,v001d,v001e \
+        --config .evolver.json \
+        [--output results.json]
+
+    python3 read_results.py \
+        --experiment v001a \
+        --config .evolver.json \
+        --format markdown
+
+Requires: pip install langsmith
+"""
+
+import argparse
+import json
+import os
+import sys
+
+
+def read_experiment(client, experiment_name):
+    """Read results from a single LangSmith experiment."""
+    try:
+        # List runs for this experiment
+        runs = list(client.list_runs(
+            project_name=experiment_name,
+            is_root=True,
+            limit=200,
+        ))
+
+        if not runs:
+            return None
+
+        per_example = {}
+        total_tokens = 0
+        total_latency_ms = 0
+        errors = 0
+
+        for run in runs:
+            example_id = str(run.reference_example_id or run.id)
+            tokens = run.total_tokens or 0
+            total_tokens += tokens
+
+            latency_ms = 0
+            if run.end_time and run.start_time:
+                latency_ms = int((run.end_time - run.start_time).total_seconds() * 1000)
+            total_latency_ms += latency_ms
+
+            has_error = bool(run.error)
+            if has_error:
+                errors += 1
+
+            # Read feedback/scores
+            feedbacks = list(client.list_feedback(run_ids=[run.id]))
+            scores = {}
+            for fb in feedbacks:
+                if fb.score is not None:
+                    scores[fb.key] = fb.score
+
+            per_example[example_id] = {
+                "score": sum(scores.values()) / len(scores) if scores else 0.0,
+                "scores": scores,
+                "tokens": tokens,
+                "latency_ms": latency_ms,
+                "error": run.error[:200] if run.error else None,
+                "input_preview": str(run.inputs)[:200] if run.inputs else "",
+                "output_preview": str(run.outputs)[:200] if run.outputs else "",
+            }
+
+        num_examples = len(per_example)
+        all_scores = [v["score"] for v in per_example.values()]
+        combined_score = sum(all_scores) / len(all_scores) if all_scores else 0.0
+
+        return {
+            "experiment": experiment_name,
+            "combined_score": combined_score,
+            "num_examples": num_examples,
+            "total_tokens": total_tokens,
+            "avg_latency_ms": total_latency_ms // max(num_examples, 1),
+            "error_count": errors,
+            "error_rate": errors / max(num_examples, 1),
+            "per_example": per_example,
+        }
+
+    except Exception as e:
+        return {"experiment": experiment_name, "error": str(e), "combined_score": 0.0}
+
+
+def compare_experiments(results_list):
+    """Compare multiple experiment results and find winner + per-task champion."""
+    if not results_list:
+        return None
+
+    valid = [r for r in results_list if "error" not in r and r.get("combined_score", 0) > 0]
+    if not valid:
+        valid = results_list
+
+    # Overall winner
+    winner = max(valid, key=lambda r: r.get("combined_score", 0))
+
+    # Per-task champion (candidate that beats winner on most individual tasks)
+    task_wins = {}
+    winner_examples = winner.get("per_example", {})
+
+    for result in valid:
+        if result["experiment"] == winner["experiment"]:
+            continue
+
+        wins = 0
+        for example_id, data in result.get("per_example", {}).items():
+            winner_score = winner_examples.get(example_id, {}).get("score", 0)
+            if data.get("score", 0) > winner_score:
+                wins += 1
+
+        if wins > 0:
+            task_wins[result["experiment"]] = wins
+
+    champion = None
+    if task_wins:
+        champion_name = max(task_wins, key=task_wins.get)
+        champion = {
+            "experiment": champion_name,
+            "task_wins": task_wins[champion_name],
+        }
+
+    return {
+        "winner": {
+            "experiment": winner["experiment"],
+            "score": winner["combined_score"],
+        },
+        "champion": champion,
+        "all_candidates": [
+            {
+                "experiment": r["experiment"],
+                "score": r.get("combined_score", 0),
+                "tokens": r.get("total_tokens", 0),
+                "latency_ms": r.get("avg_latency_ms", 0),
+                "errors": r.get("error_count", 0),
+            }
+            for r in results_list
+        ],
+    }
+
+
+def format_markdown(results):
+    """Format experiment results as markdown for agents."""
+    lines = [f"# Experiment Results: {results['experiment']}", ""]
+
+    lines.append(f"**Combined Score**: {results.get('combined_score', 0):.3f}")
+    lines.append(f"**Examples**: {results.get('num_examples', 0)}")
+    lines.append(f"**Total Tokens**: {results.get('total_tokens', 0)}")
+    lines.append(f"**Avg Latency**: {results.get('avg_latency_ms', 0)}ms")
+    lines.append(f"**Errors**: {results.get('error_count', 0)} ({results.get('error_rate', 0):.1%})")
+    lines.append("")
+
+    per_example = results.get("per_example", {})
+    if per_example:
+        # Failing examples
+        failing = {k: v for k, v in per_example.items() if v.get("score", 0) < 0.5}
+        if failing:
+            lines.append("## Failing Examples")
+            lines.append("")
+            for eid, data in sorted(failing.items(), key=lambda x: x[1].get("score", 0)):
+                lines.append(f"- **{eid}**: score={data['score']:.2f}")
+                if data.get("error"):
+                    lines.append(f"  Error: {data['error']}")
+                lines.append(f"  Input: {data.get('input_preview', 'N/A')}")
+            lines.append("")
+
+    return "\n".join(lines)
+
+
+def main():
+    parser = argparse.ArgumentParser(description="Read LangSmith experiment results")
+    parser.add_argument("--experiments", default=None, help="Comma-separated experiment names to compare")
+    parser.add_argument("--experiment", default=None, help="Single experiment to read")
+    parser.add_argument("--config", default=".evolver.json", help="Path to .evolver.json")
+    parser.add_argument("--output", default=None, help="Output JSON path")
+    parser.add_argument("--format", default="json", choices=["json", "markdown"], help="Output format")
+    args = parser.parse_args()
+
+    from langsmith import Client
+    client = Client()
+
+    if args.experiment:
+        # Single experiment
+        result = read_experiment(client, args.experiment)
+        if not result:
+            print(f"No results found for experiment: {args.experiment}", file=sys.stderr)
+            sys.exit(1)
+
+        if args.format == "markdown":
+            output = format_markdown(result)
+        else:
+            output = json.dumps(result, indent=2, default=str)
+
+        if args.output:
+            with open(args.output, "w") as f:
+                f.write(output)
+        print(output)
+
+    elif args.experiments:
+        # Compare multiple experiments
+        experiment_names = [e.strip() for e in args.experiments.split(",")]
+        results_list = []
+
+        for name in experiment_names:
+            print(f"Reading experiment: {name}...", file=sys.stderr)
+            result = read_experiment(client, name)
+            if result:
+                results_list.append(result)
+
+        if not results_list:
+            print("No experiment results found.", file=sys.stderr)
+            sys.exit(1)
+
+        comparison = compare_experiments(results_list)
+        output = json.dumps({
+            "comparison": comparison,
+            "experiments": results_list,
+        }, indent=2, default=str)
+
+        if args.output:
+            with open(args.output, "w") as f:
+                f.write(output)
+        print(output)
+
+    else:
+        print("Provide --experiment or --experiments", file=sys.stderr)
+        sys.exit(1)
+
+
+if __name__ == "__main__":
+    main()

package/tools/run_eval.py

@@ -0,0 +1,202 @@
+#!/usr/bin/env python3
+"""Run LangSmith evaluation for a candidate in a worktree.
+
+Wraps client.evaluate() — runs the user's agent against the dataset
+with configured evaluators, from within a specific directory (worktree).
+
+Usage:
+    python3 run_eval.py \
+        --config .evolver.json \
+        --worktree-path /tmp/worktree-abc \
+        --experiment-prefix v001a \
+        [--timeout 120]
+
+Requires: pip install langsmith openevals
+"""
+
+import argparse
+import json
+import os
+import subprocess
+import sys
+import tempfile
+
+
+def make_target(entry_point, cwd):
+    """Create a target function that runs the agent from a specific directory."""
+    def target(inputs):
+        input_json = json.dumps(inputs)
+        input_path = tempfile.mktemp(suffix=".json")
+        output_path = input_path + ".out"
+
+        with open(input_path, "w") as f:
+            f.write(input_json)
+
+        try:
+            cmd = entry_point
+            if "{input}" in cmd:
+                cmd = cmd.replace("{input}", input_path)
+            elif "{input_json}" in cmd:
+                cmd = cmd.replace("{input_json}", input_json)
+            else:
+                cmd = f"{cmd} --input {input_path} --output {output_path}"
+
+            env = os.environ.copy()
+            # Ensure traces go to the evolver project
+            env["LANGSMITH_TRACING"] = "true"
+
+            result = subprocess.run(
+                cmd, shell=True, capture_output=True, text=True,
+                timeout=int(os.environ.get("EVAL_TASK_TIMEOUT", "120")),
+                cwd=cwd, env=env,
+            )
+
+            # Read output file if it exists
+            if os.path.exists(output_path):
+                with open(output_path) as f:
+                    try:
+                        return json.load(f)
+                    except json.JSONDecodeError:
+                        pass
+
+            # Fallback: parse stdout
+            if result.stdout.strip():
+                try:
+                    return json.loads(result.stdout)
+                except json.JSONDecodeError:
+                    return {"output": result.stdout.strip()}
+
+            # Accept segfault (139) if output was produced
+            if result.returncode != 0:
+                return {"output": "", "error": result.stderr.strip()[:500]}
+
+            return {"output": ""}
+
+        except subprocess.TimeoutExpired:
+            return {"output": "", "error": f"TIMEOUT after {os.environ.get('EVAL_TASK_TIMEOUT', '120')}s"}
+        except Exception as e:
+            return {"output": "", "error": str(e)}
+        finally:
+            for p in [input_path, output_path]:
+                if os.path.exists(p):
+                    try:
+                        os.remove(p)
+                    except OSError:
+                        pass
+
+    return target
+
+
+def load_evaluators(evaluator_keys):
+    """Load evaluators by key name."""
+    from openevals.llm import create_llm_as_judge
+    from openevals.prompts import CORRECTNESS_PROMPT, CONCISENESS_PROMPT
+
+    evaluators = []
+    for key in evaluator_keys:
+        if key == "correctness":
+            evaluators.append(create_llm_as_judge(
+                prompt=CORRECTNESS_PROMPT,
+                feedback_key="correctness",
+                model="openai:gpt-4.1-mini",
+            ))
+        elif key == "conciseness":
+            evaluators.append(create_llm_as_judge(
+                prompt=CONCISENESS_PROMPT,
+                feedback_key="conciseness",
+                model="openai:gpt-4.1-mini",
+            ))
+        elif key == "latency":
+            def latency_eval(inputs, outputs, **kwargs):
+                return {"key": "has_output", "score": 1.0 if outputs else 0.0}
+            evaluators.append(latency_eval)
+        elif key == "token_efficiency":
+            def token_eval(inputs, outputs, **kwargs):
+                output_text = str(outputs.get("output", outputs.get("answer", "")))
+                score = min(1.0, 2000 / max(len(output_text), 1))
+                return {"key": "token_efficiency", "score": score}
+            evaluators.append(token_eval)
+
+    return evaluators
+
+
+def main():
+    parser = argparse.ArgumentParser(description="Run LangSmith evaluation for a candidate")
+    parser.add_argument("--config", default=".evolver.json", help="Path to .evolver.json")
+    parser.add_argument("--worktree-path", required=True, help="Path to the candidate's worktree")
+    parser.add_argument("--experiment-prefix", required=True, help="Experiment name prefix (e.g. v001a)")
+    parser.add_argument("--timeout", type=int, default=120, help="Per-task timeout in seconds")
+    args = parser.parse_args()
+
+    with open(args.config) as f:
+        config = json.load(f)
+
+    os.environ["EVAL_TASK_TIMEOUT"] = str(args.timeout)
+
+    from langsmith import Client
+    client = Client()
+
+    target = make_target(config["entry_point"], args.worktree_path)
+    evaluators = load_evaluators(config["evaluators"])
+
+    print(f"Running evaluation: {args.experiment_prefix}")
+    print(f"  Dataset: {config['dataset']}")
+    print(f"  Worktree: {args.worktree_path}")
+    print(f"  Evaluators: {config['evaluators']}")
+
+    try:
+        results = client.evaluate(
+            target,
+            data=config["dataset"],
+            evaluators=evaluators,
+            experiment_prefix=args.experiment_prefix,
+            max_concurrency=1,
+        )
+
+        experiment_name = results.experiment_name
+
+        # Calculate mean score
+        scores = []
+        per_example = {}
+        for result in results:
+            example_scores = []
+            if result.evaluation_results and result.evaluation_results.get("results"):
+                for er in result.evaluation_results["results"]:
+                    if er.get("score") is not None:
+                        example_scores.append(er["score"])
+                        scores.append(er["score"])
+
+            example_id = str(result.example.id) if result.example else "unknown"
+            per_example[example_id] = {
+                "score": sum(example_scores) / len(example_scores) if example_scores else 0.0,
+                "num_evaluators": len(example_scores),
+            }
+
+        mean_score = sum(scores) / len(scores) if scores else 0.0
+
+        output = {
+            "experiment": experiment_name,
+            "prefix": args.experiment_prefix,
+            "combined_score": mean_score,
+            "num_examples": len(per_example),
+            "num_scores": len(scores),
+            "per_example": per_example,
+        }
+
+        print(json.dumps(output))
+        print(f"\nEvaluation complete: {mean_score:.3f} ({len(per_example)} examples)")
+
+    except Exception as e:
+        print(f"Evaluation failed: {e}", file=sys.stderr)
+        output = {
+            "experiment": None,
+            "prefix": args.experiment_prefix,
+            "combined_score": 0.0,
+            "error": str(e),
+        }
+        print(json.dumps(output))
+        sys.exit(1)
+
+
+if __name__ == "__main__":
+    main()