harness-evolver 1.7.0 → 1.9.0

package/agents/harness-evolver-proposer.md

@@ -13,6 +13,16 @@ permissionMode: acceptEdits
 If your prompt contains a `<files_to_read>` block, you MUST use the Read tool to load
 every file listed there before performing any other actions. These files are your context.
 
+## Strategy Injection
+
+Your prompt may contain a `<strategy>` block defining your evolutionary role:
+- **exploitation**: Make targeted, conservative fixes to the current best
+- **exploration**: Try fundamentally different approaches, be bold
+- **crossover**: Combine strengths from two parent versions
+
+Follow the strategy. It determines your risk tolerance and parent selection.
+If no strategy block is present, default to exploitation (conservative improvement).
+
 ## Context7 — Enrich Your Knowledge
 
 You have access to Context7 MCP tools (`resolve-library-id` and `get-library-docs`) for looking up **current, version-specific documentation** of any library.

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "harness-evolver",
-  "version": "1.7.0",
+  "version": "1.9.0",
   "description": "Meta-Harness-style autonomous harness optimization for Claude Code",
   "author": "Raphael Valdetaro",
   "license": "MIT",

package/skills/evolve/SKILL.md

@@ -36,97 +36,215 @@ python3 -c "import json; s=json.load(open('.harness-evolver/summary.json')); pri
 
 ### 1.5. Gather LangSmith Traces (MANDATORY after every evaluation)
 
-**Run these commands unconditionally after EVERY evaluation** (including baseline). If langsmith-cli is not installed or there are no runs, the commands fail silently — that's fine. But you MUST attempt them.
+**Run these commands unconditionally after EVERY evaluation** (including baseline). Do NOT guess project names — discover them.
+
+**Step 1: Find the actual LangSmith project name**
 
 ```bash
-langsmith-cli --json runs list --project harness-evolver-{last_evaluated_version} --failed --fields id,name,error,inputs --limit 10 > .harness-evolver/langsmith_diagnosis.json 2>/dev/null || echo "[]" > .harness-evolver/langsmith_diagnosis.json
+langsmith-cli --json projects list --name-pattern "harness-evolver*" --limit 10 2>/dev/null
+```
+
+This returns all projects matching the prefix. Pick the most recently updated one, or the one matching the current version. Save the project name:
 
-langsmith-cli --json runs stats --project harness-evolver-{last_evaluated_version} > .harness-evolver/langsmith_stats.json 2>/dev/null || echo "{}" > .harness-evolver/langsmith_stats.json
+```bash
+LS_PROJECT=$(langsmith-cli --json projects list --name-pattern "harness-evolver*" --limit 1 2>/dev/null | python3 -c "import sys,json; data=json.load(sys.stdin); print(data[0]['name'] if data else '')" 2>/dev/null || echo "")
 ```
 
-For the first iteration, use `baseline` as the version. For subsequent iterations, use the latest evaluated version.
+If `LS_PROJECT` is empty, langsmith-cli is not available or no projects exist — skip to step 2.
+
+**Step 2: Gather traces from the discovered project**
+
+```bash
+if [ -n "$LS_PROJECT" ]; then
+  langsmith-cli --json runs list --project "$LS_PROJECT" --failed --fields id,name,error,inputs --limit 10 > .harness-evolver/langsmith_diagnosis.json 2>/dev/null || echo "[]" > .harness-evolver/langsmith_diagnosis.json
+  langsmith-cli --json runs stats --project "$LS_PROJECT" > .harness-evolver/langsmith_stats.json 2>/dev/null || echo "{}" > .harness-evolver/langsmith_stats.json
+  echo "$LS_PROJECT" > .harness-evolver/langsmith_project.txt
+else
+  echo "[]" > .harness-evolver/langsmith_diagnosis.json
+  echo "{}" > .harness-evolver/langsmith_stats.json
+fi
+```
 
 These files are included in the proposer's `<files_to_read>` so it has real trace data for diagnosis.
 
-### 2. Propose
+### 2. Propose (3 parallel candidates)
 
-Dispatch a subagent using the **Agent tool**.
+Spawn 3 proposer agents IN PARALLEL, each with a different evolutionary strategy.
+This follows the DGM/AlphaEvolve pattern: exploit + explore + crossover.
 
-First, read the proposer agent definition to include in the prompt:
+First, read the proposer agent definition:
 ```bash
 cat ~/.claude/agents/harness-evolver-proposer.md
 ```
 
-Then dispatch the Agent with the agent definition + structured task:
+Then determine parents for each strategy:
+- **Exploiter parent**: current best version (from summary.json `best.version`)
+- **Explorer parent**: a non-best version with low offspring count (read summary.json history, pick one that scored >0 but is NOT the best and has NOT been parent to many children)
+- **Crossover parents**: best version + a different high-scorer from a different lineage
+
+Spawn all 3 using the Agent tool. The first 2 use `run_in_background: true`, the 3rd blocks:
 
+**Candidate A (Exploiter)** — `run_in_background: true`:
 ```
 Agent(
-  description: "Propose harness {version}",
+  description: "Proposer A (exploit): targeted fix for {version}",
+  run_in_background: true,
   prompt: |
     <agent_instructions>
-    {paste the FULL content of harness-evolver-proposer.md here}
+    {FULL content of harness-evolver-proposer.md}
     </agent_instructions>
 
+    <strategy>
+    APPROACH: exploitation
+    You are the EXPLOITER. Make the SMALLEST, most targeted change that fixes
+    the highest-impact failing tasks. Base your work on the current best version.
+    Do NOT restructure the code. Do NOT change the architecture.
+    Focus on: prompt tweaks, parameter tuning, fixing specific failure modes.
+    </strategy>
+
     <objective>
-    Propose harness version {version} that improves on the current best score of {best_score}.
+    Propose harness version {version}a that improves on {best_score}.
     </objective>
 
     <files_to_read>
     - .harness-evolver/summary.json
     - .harness-evolver/PROPOSER_HISTORY.md
     - .harness-evolver/config.json
-    - .harness-evolver/baseline/harness.py
     - .harness-evolver/harnesses/{best_version}/harness.py
     - .harness-evolver/harnesses/{best_version}/scores.json
     - .harness-evolver/harnesses/{best_version}/proposal.md
-    - .harness-evolver/langsmith_diagnosis.json (if exists — LangSmith failure analysis)
-    - .harness-evolver/langsmith_stats.json (if exists — LangSmith aggregate stats)
-    - .harness-evolver/architecture.json (if exists — architect topology recommendation)
+    - .harness-evolver/langsmith_diagnosis.json (if exists)
+    - .harness-evolver/langsmith_stats.json (if exists)
+    - .harness-evolver/architecture.json (if exists)
     </files_to_read>
 
     <output>
-    Create directory .harness-evolver/harnesses/{version}/ containing:
-    - harness.py (the improved harness)
-    - config.json (parameters, copy from parent if unchanged)
-    - proposal.md (reasoning, must start with "Based on v{PARENT}")
+    Create directory .harness-evolver/harnesses/{version}a/ containing:
+    - harness.py, config.json, proposal.md
     </output>
+)
+```
 
-    <success_criteria>
-    - harness.py maintains CLI interface (--input, --output, --traces-dir, --config)
-    - proposal.md documents evidence-based reasoning
-    - If proposing API changes, MUST use Context7 (resolve-library-id + get-library-docs) to verify current docs
-    - Changes motivated by LangSmith trace data (in langsmith_diagnosis.json) when available
-    </success_criteria>
+**Candidate B (Explorer)** — `run_in_background: true`:
+```
+Agent(
+  description: "Proposer B (explore): bold change from {explorer_parent}",
+  run_in_background: true,
+  prompt: |
+    <agent_instructions>
+    {FULL content of harness-evolver-proposer.md}
+    </agent_instructions>
+
+    <strategy>
+    APPROACH: exploration
+    You are the EXPLORER. Try a FUNDAMENTALLY DIFFERENT approach.
+    Base your work on {explorer_parent} (NOT the current best — intentionally diverging).
+    Consider: different retrieval strategy, different prompt structure,
+    different output parsing, different error handling philosophy.
+    Be bold. A creative failure teaches more than a timid success.
+    </strategy>
+
+    <objective>
+    Propose harness version {version}b that takes a different approach.
+    </objective>
+
+    <files_to_read>
+    - .harness-evolver/summary.json
+    - .harness-evolver/PROPOSER_HISTORY.md
+    - .harness-evolver/config.json
+    - .harness-evolver/baseline/harness.py
+    - .harness-evolver/harnesses/{explorer_parent}/harness.py
+    - .harness-evolver/harnesses/{explorer_parent}/scores.json
+    - .harness-evolver/langsmith_diagnosis.json (if exists)
+    - .harness-evolver/architecture.json (if exists)
+    </files_to_read>
+
+    <output>
+    Create directory .harness-evolver/harnesses/{version}b/ containing:
+    - harness.py, config.json, proposal.md
+    </output>
 )
 ```
 
-Wait for `## PROPOSAL COMPLETE` in the response.
+**Candidate C (Crossover)** — blocks (last one):
+```
+Agent(
+  description: "Proposer C (crossover): combine {parent_a} + {parent_b}",
+  prompt: |
+    <agent_instructions>
+    {FULL content of harness-evolver-proposer.md}
+    </agent_instructions>
+
+    <strategy>
+    APPROACH: crossover
+    You are the CROSSOVER agent. Combine the STRENGTHS of two different versions:
+    - {parent_a} (score: {score_a}): {summary of what it does well}
+    - {parent_b} (score: {score_b}): {summary of what it does well}
+    Take the best elements from each and merge them into a single harness.
+    </strategy>
+
+    <objective>
+    Propose harness version {version}c that combines the best of {parent_a} and {parent_b}.
+    </objective>
+
+    <files_to_read>
+    - .harness-evolver/summary.json
+    - .harness-evolver/PROPOSER_HISTORY.md
+    - .harness-evolver/config.json
+    - .harness-evolver/harnesses/{parent_a}/harness.py
+    - .harness-evolver/harnesses/{parent_a}/scores.json
+    - .harness-evolver/harnesses/{parent_b}/harness.py
+    - .harness-evolver/harnesses/{parent_b}/scores.json
+    - .harness-evolver/langsmith_diagnosis.json (if exists)
+    - .harness-evolver/architecture.json (if exists)
+    </files_to_read>
+
+    <output>
+    Create directory .harness-evolver/harnesses/{version}c/ containing:
+    - harness.py, config.json, proposal.md
+    </output>
+)
+```
+
+Wait for all 3 to complete. The background agents will notify when done.
+
+**Special case — iteration 1**: Only the exploiter and explorer can run (no second parent for crossover yet). Spawn 2 agents: exploiter (from baseline) and explorer (also from baseline but with bold strategy). Skip crossover.
 
-### 3. Validate
+**Special case — iteration 2+**: All 3 strategies. Explorer parent = fitness-weighted random from history excluding current best.
 
+### 3. Validate All Candidates
+
+For each candidate (a, b, c):
 ```bash
-python3 $TOOLS/evaluate.py validate \
-    --harness .harness-evolver/harnesses/{version}/harness.py \
-    --config .harness-evolver/harnesses/{version}/config.json
+python3 $TOOLS/evaluate.py validate --harness .harness-evolver/harnesses/{version}{suffix}/harness.py --config .harness-evolver/harnesses/{version}{suffix}/config.json
 ```
 
-If fails: one retry via proposer. If still fails: score 0.0, continue.
+Remove any that fail validation.
 
-### 4. Evaluate
+### 4. Evaluate All Candidates
 
+For each valid candidate:
 ```bash
 python3 $TOOLS/evaluate.py run \
-    --harness .harness-evolver/harnesses/{version}/harness.py \
-    --config .harness-evolver/harnesses/{version}/config.json \
+    --harness .harness-evolver/harnesses/{version}{suffix}/harness.py \
+    --config .harness-evolver/harnesses/{version}{suffix}/config.json \
     --tasks-dir .harness-evolver/eval/tasks/ \
     --eval .harness-evolver/eval/eval.py \
-    --traces-dir .harness-evolver/harnesses/{version}/traces/ \
-    --scores .harness-evolver/harnesses/{version}/scores.json \
+    --traces-dir .harness-evolver/harnesses/{version}{suffix}/traces/ \
+    --scores .harness-evolver/harnesses/{version}{suffix}/scores.json \
     --timeout 60
 ```
 
-### 5. Update State
+### 5. Select Winner + Update State
 
+Compare scores of all evaluated candidates. The winner is the one with highest combined_score.
+
+Rename the winner directory to the official version name:
+```bash
+mv .harness-evolver/harnesses/{version}{winning_suffix} .harness-evolver/harnesses/{version}
+```
+
+Update state with the winner:
 ```bash
 python3 $TOOLS/state.py update \
     --base-dir .harness-evolver \
@@ -135,6 +253,17 @@ python3 $TOOLS/state.py update \
     --proposal .harness-evolver/harnesses/{version}/proposal.md
 ```
 
+Report ALL candidates:
+```
+Iteration {i}/{N} — 3 candidates evaluated:
+  {version}a (exploit): {score_a} — {1-line summary from proposal.md}
+  {version}b (explore): {score_b} — {1-line summary}
+  {version}c (cross):   {score_c} — {1-line summary}
+  Winner: {version}{suffix} ({score}) ← promoted to {version}
+```
+
+Keep losing candidates in their directories (they're part of the archive — never discard, per DGM).
+
 ### 6. Report
 
 Read `summary.json`. Print: `Iteration {i}/{N}: {version} scored {score} (best: {best} at {best_score})`

package/tools/evaluate.py

@@ -118,12 +118,17 @@ def cmd_run(args):
             api_key = os.environ.get(ls.get("api_key_env", "LANGSMITH_API_KEY"), "")
             if api_key:
                 version = os.path.basename(os.path.dirname(traces_dir))
+                ls_project = f"{ls.get('project_prefix', 'harness-evolver')}-{version}"
                 langsmith_env = {
                     **os.environ,
                     "LANGCHAIN_TRACING_V2": "true",
                     "LANGCHAIN_API_KEY": api_key,
-                    "LANGCHAIN_PROJECT": f"{ls.get('project_prefix', 'harness-evolver')}-{version}",
+                    "LANGCHAIN_PROJECT": ls_project,
                 }
+                # Write the project name so the evolve skill knows where to find traces
+                ls_project_file = os.path.join(os.path.dirname(os.path.dirname(traces_dir)), "langsmith_project.txt")
+                with open(ls_project_file, "w") as f:
+                    f.write(ls_project)
 
     for task_file in task_files:
         task_path = os.path.join(tasks_dir, task_file)