harness-evolver 1.3.0 → 1.4.0

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "harness-evolver",
-  "version": "1.3.0",
+  "version": "1.4.0",
   "description": "Meta-Harness-style autonomous harness optimization for Claude Code",
   "author": "Raphael Valdetaro",
   "license": "MIT",

package/skills/evolve/SKILL.md

@@ -5,7 +5,7 @@ argument-hint: "[--iterations N]"
 allowed-tools: [Read, Write, Edit, Bash, Glob, Grep, Agent]
 ---
 
-# /harness-evolve
+# /harness-evolver:evolve
 
 Run the autonomous propose-evaluate-iterate loop.
 
@@ -34,9 +34,45 @@ For each iteration:
 python3 -c "import json; s=json.load(open('.harness-evolver/summary.json')); print(f'v{s[\"iterations\"]+1:03d}')"
 ```
 
+### 1.5. Gather Diagnostic Context (LangSmith + Context7)
+
+**This step is MANDATORY before every propose.** The orchestrator gathers data so the proposer receives it as files.
+
+**LangSmith (if enabled):**
+
+Check if LangSmith is enabled and langsmith-cli is available:
+```bash
+cat .harness-evolver/config.json | python3 -c "import sys,json; print(json.load(sys.stdin).get('eval',{}).get('langsmith',{}).get('enabled',False))"
+which langsmith-cli 2>/dev/null
+```
+
+If BOTH are true AND at least one iteration has run, gather LangSmith data:
+```bash
+langsmith-cli --json runs list --project harness-evolver-{best_version} --failed --fields id,name,error,inputs --limit 10 > .harness-evolver/langsmith_diagnosis.json 2>/dev/null || echo "[]" > .harness-evolver/langsmith_diagnosis.json
+
+langsmith-cli --json runs stats --project harness-evolver-{best_version} > .harness-evolver/langsmith_stats.json 2>/dev/null || echo "{}" > .harness-evolver/langsmith_stats.json
+```
+
+**Context7 (if available):**
+
+Check `config.json` field `stack.detected`. For each detected library, use the Context7 MCP tools to fetch relevant documentation:
+
+```
+For each library in stack.detected:
+  1. resolve-library-id with the context7_id
+  2. get-library-docs with a query relevant to the current failure modes
+  3. Save output to .harness-evolver/context7_docs.md (append each library's docs)
+```
+
+This runs ONCE per iteration, not per library. Focus on the library most relevant to the current failures.
+
+If Context7 MCP is not available, skip silently.
+
 ### 2. Propose
 
-Spawn the `harness-evolver-proposer` agent with a structured prompt:
+Spawn the `harness-evolver-proposer` agent with a structured prompt.
+
+The `<files_to_read>` MUST include the LangSmith/Context7 files if they were gathered:
 
 ```xml
 <objective>
@@ -51,6 +87,10 @@ Propose harness version {version} that improves on the current best score of {be
 - .harness-evolver/harnesses/{best_version}/harness.py
 - .harness-evolver/harnesses/{best_version}/scores.json
 - .harness-evolver/harnesses/{best_version}/proposal.md
+- .harness-evolver/langsmith_diagnosis.json (if exists — LangSmith failure analysis)
+- .harness-evolver/langsmith_stats.json (if exists — LangSmith aggregate stats)
+- .harness-evolver/context7_docs.md (if exists — current library documentation)
+- .harness-evolver/architecture.json (if exists — architect topology recommendation)
 </files_to_read>
 
 <output>
@@ -63,7 +103,8 @@ Create directory .harness-evolver/harnesses/{version}/ containing:
 <success_criteria>
 - harness.py maintains CLI interface (--input, --output, --traces-dir, --config)
 - proposal.md documents evidence-based reasoning
-- Changes are motivated by trace analysis, not guesswork
+- Changes are motivated by trace analysis (LangSmith data if available), not guesswork
+- If context7_docs.md was provided, API usage must match current documentation
 </success_criteria>
 ```
 
@@ -106,26 +147,73 @@ python3 $TOOLS/state.py update \
 
 Read `summary.json`. Print: `Iteration {i}/{N}: {version} scored {score} (best: {best} at {best_score})`
 
-### 6.5. Check for Eval Gaming
+### 6.5. Auto-trigger Critic (on eval gaming)
 
-After updating state, read the latest `summary.json` and check:
+Read `summary.json` and check:
 - Did the score jump >0.3 from parent version?
 - Did we reach 1.0 in fewer than 3 total iterations?
 
-If either is true, warn:
+If EITHER is true, **AUTO-SPAWN the critic agent** (do not just suggest — actually spawn it):
+
+```bash
+python3 $TOOLS/evaluate.py run \
+    --harness .harness-evolver/harnesses/{version}/harness.py \
+    --tasks-dir .harness-evolver/eval/tasks/ \
+    --eval .harness-evolver/eval/eval.py \
+    --traces-dir /tmp/critic-check/ \
+    --scores /tmp/critic-check-scores.json \
+    --timeout 60
+```
+
+Spawn the `harness-evolver-critic` agent:
 
-> Suspicious convergence detected: score jumped from {parent_score} to {score} in one iteration.
-> The eval may be too lenient. Run `/harness-evolver:critic` to analyze eval quality.
+```xml
+<objective>
+EVAL GAMING DETECTED: Score jumped from {parent_score} to {score} in one iteration.
+Analyze the eval quality and propose a stricter eval.
+</objective>
 
-If score is 1.0 and iterations < 3, STOP the loop and strongly recommend the critic:
+<files_to_read>
+- .harness-evolver/eval/eval.py
+- .harness-evolver/summary.json
+- .harness-evolver/harnesses/{version}/scores.json
+- .harness-evolver/harnesses/{version}/harness.py
+- .harness-evolver/harnesses/{version}/proposal.md
+- .harness-evolver/config.json
+- .harness-evolver/langsmith_stats.json (if exists)
+</files_to_read>
 
-> Perfect score reached in only {iterations} iteration(s). This usually indicates
-> the eval is too easy, not that the harness is perfect. Run `/harness-evolver:critic`
-> before continuing.
+<output>
+Write:
+- .harness-evolver/critic_report.md
+- .harness-evolver/eval/eval_improved.py (if weaknesses found)
+</output>
+
+<success_criteria>
+- Identifies specific weaknesses in eval.py with task/output examples
+- If gaming detected, shows exact tasks that expose the weakness
+- Improved eval preserves the --results-dir/--tasks-dir/--scores interface
+- Re-scores the best version with improved eval to show the difference
+</success_criteria>
+```
+
+Wait for `## CRITIC REPORT COMPLETE`.
+
+If critic wrote `eval_improved.py`:
+- Re-score the best harness with the improved eval
+- Show the score difference (e.g., "Current eval: 1.0. Improved eval: 0.45")
+- **AUTO-ADOPT the improved eval**: copy `eval_improved.py` to `eval/eval.py`
+- Re-run baseline with new eval and update `summary.json`
+- Print: "Eval upgraded. Resuming evolution with stricter eval."
+- **Continue the loop** with the new eval
+
+If critic did NOT write `eval_improved.py` (eval is fine):
+- Print the critic's assessment
+- Continue the loop normally
 
 ### 7. Auto-trigger Architect (on stagnation or regression)
 
-Check if the architect should be auto-spawned. This happens when:
+Check if the architect should be auto-spawned:
 - **Stagnation**: 3 consecutive iterations within 1% of each other
 - **Regression**: score dropped below parent score (even once)
 
@@ -141,7 +229,7 @@ python3 $TOOLS/analyze_architecture.py \
     -o .harness-evolver/architecture_signals.json
 ```
 
-Then spawn the `harness-evolver-architect` agent:
+Spawn the `harness-evolver-architect` agent:
 
 ```xml
 <objective>
@@ -156,6 +244,7 @@ Analyze the harness architecture and recommend a topology change.
 - .harness-evolver/config.json
 - .harness-evolver/harnesses/{best_version}/harness.py
 - .harness-evolver/harnesses/{best_version}/scores.json
+- .harness-evolver/context7_docs.md (if exists)
 </files_to_read>
 
 <output>
@@ -173,25 +262,21 @@ Write:
 
 Wait for `## ARCHITECTURE ANALYSIS COMPLETE`.
 
-After the architect completes, report:
-
-> Architect recommends: {current} → {recommended} ({confidence} confidence)
-> Migration path: {N} steps. Continuing evolution with architecture guidance.
-
-Then **continue the loop** — the proposer will read `architecture.json` in the next iteration.
+Report: `Architect recommends: {current} → {recommended} ({confidence} confidence)`
 
-If `architecture.json` already exists (architect already ran), skip — don't re-run.
+Then **continue the loop** — the proposer reads `architecture.json` in the next iteration.
 
 ### 8. Check Stop Conditions
 
 - **Target**: `combined_score >= target_score` → stop
 - **N reached**: done
-- **Stagnation post-architect**: 3 more iterations without improvement AFTER architect ran → stop (architecture change didn't help)
+- **Stagnation post-architect**: 3 more iterations without improvement AFTER architect ran → stop
 
 ## When Loop Ends — Final Report
 
 - Best version and score
 - Improvement over baseline (absolute and %)
 - Total iterations run
+- Whether critic was triggered and eval was upgraded
 - Whether architect was triggered and what it recommended
 - Suggest: "The best harness is at `.harness-evolver/harnesses/{best}/harness.py`. Copy it to your project."