harness-evolver 1.8.0 → 2.0.0

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "harness-evolver",
-  "version": "1.8.0",
+  "version": "2.0.0",
   "description": "Meta-Harness-style autonomous harness optimization for Claude Code",
   "author": "Raphael Valdetaro",
   "license": "MIT",

package/skills/evolve/SKILL.md

@@ -36,15 +36,34 @@ python3 -c "import json; s=json.load(open('.harness-evolver/summary.json')); pri
 
 ### 1.5. Gather LangSmith Traces (MANDATORY after every evaluation)
 
-**Run these commands unconditionally after EVERY evaluation** (including baseline). If langsmith-cli is not installed or there are no runs, the commands fail silently — that's fine. But you MUST attempt them.
+**Run these commands unconditionally after EVERY evaluation** (including baseline). Do NOT guess project names — discover them.
+
+**Step 1: Find the actual LangSmith project name**
 
 ```bash
-langsmith-cli --json runs list --project harness-evolver-{last_evaluated_version} --failed --fields id,name,error,inputs --limit 10 > .harness-evolver/langsmith_diagnosis.json 2>/dev/null || echo "[]" > .harness-evolver/langsmith_diagnosis.json
+langsmith-cli --json projects list --name-pattern "harness-evolver*" --limit 10 2>/dev/null
+```
 
-langsmith-cli --json runs stats --project harness-evolver-{last_evaluated_version} > .harness-evolver/langsmith_stats.json 2>/dev/null || echo "{}" > .harness-evolver/langsmith_stats.json
+This returns all projects matching the prefix. Pick the most recently updated one, or the one matching the current version. Save the project name:
+
+```bash
+LS_PROJECT=$(langsmith-cli --json projects list --name-pattern "harness-evolver*" --limit 1 2>/dev/null | python3 -c "import sys,json; data=json.load(sys.stdin); print(data[0]['name'] if data else '')" 2>/dev/null || echo "")
 ```
 
-For the first iteration, use `baseline` as the version. For subsequent iterations, use the latest evaluated version.
+If `LS_PROJECT` is empty, langsmith-cli is not available or no projects exist — skip to step 2.
+
+**Step 2: Gather traces from the discovered project**
+
+```bash
+if [ -n "$LS_PROJECT" ]; then
+  langsmith-cli --json runs list --project "$LS_PROJECT" --failed --fields id,name,error,inputs --limit 10 > .harness-evolver/langsmith_diagnosis.json 2>/dev/null || echo "[]" > .harness-evolver/langsmith_diagnosis.json
+  langsmith-cli --json runs stats --project "$LS_PROJECT" > .harness-evolver/langsmith_stats.json 2>/dev/null || echo "{}" > .harness-evolver/langsmith_stats.json
+  echo "$LS_PROJECT" > .harness-evolver/langsmith_project.txt
+else
+  echo "[]" > .harness-evolver/langsmith_diagnosis.json
+  echo "{}" > .harness-evolver/langsmith_stats.json
+fi
+```
 
 These files are included in the proposer's `<files_to_read>` so it has real trace data for diagnosis.
 
@@ -187,15 +206,41 @@ Agent(
 )
 ```
 
-Wait for all 3 to complete. The background agents will notify when done.
+**Also spawn these additional candidates:**
+
+**Candidate D (Prompt Specialist)** — `run_in_background: true`:
+Same as Exploiter but with a different focus:
+```
+<strategy>
+APPROACH: prompt-engineering
+You are the PROMPT SPECIALIST. Focus ONLY on improving the system prompt,
+few-shot examples, output format instructions, and prompt structure.
+Do NOT change the retrieval logic, pipeline structure, or code architecture.
+</strategy>
+```
+Output to: `.harness-evolver/harnesses/{version}d/`
+
+**Candidate E (Data/Retrieval Specialist)** — `run_in_background: true`:
+```
+<strategy>
+APPROACH: retrieval-optimization  
+You are the RETRIEVAL SPECIALIST. Focus ONLY on improving how data is
+retrieved, filtered, ranked, and presented to the LLM. 
+Do NOT change the system prompt text or output formatting.
+Improve: search logic, relevance scoring, cross-domain retrieval, chunking.
+</strategy>
+```
+Output to: `.harness-evolver/harnesses/{version}e/`
+
+Wait for all 5 to complete. The background agents will notify when done.
 
-**Special case — iteration 1**: Only the exploiter and explorer can run (no second parent for crossover yet). Spawn 2 agents: exploiter (from baseline) and explorer (also from baseline but with bold strategy). Skip crossover.
+**Minimum 3 candidates ALWAYS, even on iteration 1.** On iteration 1, the crossover agent uses baseline as both parents but with instruction to "combine the best retrieval strategy with the best prompt strategy from your analysis of the baseline." On iteration 2+, crossover uses two genuinely different parents.
 
-**Special case — iteration 2+**: All 3 strategies. Explorer parent = fitness-weighted random from history excluding current best.
+**On iteration 3+**: If scores are improving, keep all 5 strategies. If stagnating, replace Candidate D with a "Radical" strategy that rewrites the harness from scratch.
 
 ### 3. Validate All Candidates
 
-For each candidate (a, b, c):
+For each candidate (a, b, c, d, e):
 ```bash
 python3 $TOOLS/evaluate.py validate --harness .harness-evolver/harnesses/{version}{suffix}/harness.py --config .harness-evolver/harnesses/{version}{suffix}/config.json
 ```

package/tools/evaluate.py

@@ -118,12 +118,17 @@ def cmd_run(args):
             api_key = os.environ.get(ls.get("api_key_env", "LANGSMITH_API_KEY"), "")
             if api_key:
                 version = os.path.basename(os.path.dirname(traces_dir))
+                ls_project = f"{ls.get('project_prefix', 'harness-evolver')}-{version}"
                 langsmith_env = {
                     **os.environ,
                     "LANGCHAIN_TRACING_V2": "true",
                     "LANGCHAIN_API_KEY": api_key,
-                    "LANGCHAIN_PROJECT": f"{ls.get('project_prefix', 'harness-evolver')}-{version}",
+                    "LANGCHAIN_PROJECT": ls_project,
                 }
+                # Write the project name so the evolve skill knows where to find traces
+                ls_project_file = os.path.join(os.path.dirname(os.path.dirname(traces_dir)), "langsmith_project.txt")
+                with open(ls_project_file, "w") as f:
+                    f.write(ls_project)
 
     for task_file in task_files:
         task_path = os.path.join(tasks_dir, task_file)