harness-evolver 1.5.0 → 1.7.0

package/agents/harness-evolver-architect.md

@@ -4,6 +4,7 @@ description: |
   Use this agent to analyze harness architecture and recommend optimal multi-agent topology.
   Reads code analysis signals, traces, and scores to produce a migration plan.
 tools: Read, Write, Bash, Grep, Glob
+color: blue
 ---
 
 ## Bootstrap

package/agents/harness-evolver-critic.md

@@ -4,6 +4,7 @@ description: |
   Use this agent to assess eval quality, detect eval gaming, and propose stricter evaluation.
   Triggered when scores converge suspiciously fast or on user request.
 tools: Read, Write, Bash, Grep, Glob
+color: red
 ---
 
 ## Bootstrap

package/agents/harness-evolver-proposer.md

@@ -4,6 +4,7 @@ description: |
   Use this agent when the evolve skill needs to propose a new harness candidate.
   Navigates the .harness-evolver/ filesystem to diagnose failures and propose improvements.
 tools: Read, Write, Edit, Bash, Glob, Grep
+color: green
 permissionMode: acceptEdits
 ---
 
@@ -12,6 +13,24 @@ permissionMode: acceptEdits
 If your prompt contains a `<files_to_read>` block, you MUST use the Read tool to load
 every file listed there before performing any other actions. These files are your context.
 
+## Context7 — Enrich Your Knowledge
+
+You have access to Context7 MCP tools (`resolve-library-id` and `get-library-docs`) for looking up **current, version-specific documentation** of any library.
+
+**USE CONTEXT7 PROACTIVELY whenever you:**
+- Are about to write code that uses a library API (LangGraph, LangChain, OpenAI, etc.)
+- Are unsure about the correct method signature, parameters, or patterns
+- Want to check if a better approach exists in the latest version
+- See an error in traces that might be caused by using a deprecated API
+
+**How to use:**
+1. `resolve-library-id` with the library name (e.g., "langchain", "langgraph")
+2. `get-library-docs` with a specific query (e.g., "StateGraph conditional edges", "ChatGoogleGenerativeAI streaming")
+
+**Do NOT skip this.** Your training data may be outdated. Context7 gives you the current docs. Even if you're confident about an API, a quick check takes seconds and prevents proposing deprecated patterns.
+
+If Context7 is not available, proceed with model knowledge but note in `proposal.md`: "API not verified against current docs."
+
 ## Return Protocol
 
 When done, end your response with:

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "harness-evolver",
-  "version": "1.5.0",
+  "version": "1.7.0",
   "description": "Meta-Harness-style autonomous harness optimization for Claude Code",
   "author": "Raphael Valdetaro",
   "license": "MIT",

package/skills/architect/SKILL.md

@@ -48,13 +48,21 @@ python3 $TOOLS/analyze_architecture.py \
     -o .harness-evolver/architecture_signals.json
 ```
 
-3. Dispatch subagent using the **Agent tool** with `subagent_type: "harness-evolver-architect"`:
+3. Read the architect agent definition:
+```bash
+cat ~/.claude/agents/harness-evolver-architect.md
+```
+
+4. Dispatch using the Agent tool — include the agent definition in the prompt:
 
 ```
 Agent(
-  subagent_type: "harness-evolver-architect",
   description: "Architect: topology analysis",
   prompt: |
+    <agent_instructions>
+    {paste the FULL content of harness-evolver-architect.md here}
+    </agent_instructions>
+
     <objective>
     Analyze the harness architecture and recommend the optimal multi-agent topology.
     {If called from evolve: "The evolution loop stagnated/regressed after N iterations."}

package/skills/critic/SKILL.md

@@ -22,13 +22,21 @@ TOOLS=$([ -d ".harness-evolver/tools" ] && echo ".harness-evolver/tools" || echo
 
 1. Read `summary.json` and identify the suspicious pattern (score jump, premature convergence).
 
-2. Dispatch subagent using the **Agent tool** with `subagent_type: "harness-evolver-critic"`:
+2. Read the critic agent definition:
+```bash
+cat ~/.claude/agents/harness-evolver-critic.md
+```
+
+3. Dispatch using the Agent tool — include the agent definition in the prompt:
 
 ```
 Agent(
-  subagent_type: "harness-evolver-critic",
   description: "Critic: analyze eval quality",
   prompt: |
+    <agent_instructions>
+    {paste the FULL content of harness-evolver-critic.md here}
+    </agent_instructions>
+
     <objective>
     Analyze eval quality for this harness evolution project.
     The best version is {version} with score {score} achieved in {iterations} iteration(s).

package/skills/evolve/SKILL.md

@@ -34,51 +34,39 @@ For each iteration:
 python3 -c "import json; s=json.load(open('.harness-evolver/summary.json')); print(f'v{s[\"iterations\"]+1:03d}')"
 ```
 
-### 1.5. Gather Diagnostic Context (LangSmith + Context7)
+### 1.5. Gather LangSmith Traces (MANDATORY after every evaluation)
 
-**This step is MANDATORY before every propose.** The orchestrator gathers data so the proposer receives it as files.
+**Run these commands unconditionally after EVERY evaluation** (including baseline). If langsmith-cli is not installed or there are no runs, the commands fail silently — that's fine. But you MUST attempt them.
 
-**LangSmith (if enabled):**
-
-Check if LangSmith is enabled and langsmith-cli is available:
-```bash
-cat .harness-evolver/config.json | python3 -c "import sys,json; print(json.load(sys.stdin).get('eval',{}).get('langsmith',{}).get('enabled',False))"
-which langsmith-cli 2>/dev/null
-```
-
-If BOTH are true AND at least one iteration has run, gather LangSmith data:
 ```bash
-langsmith-cli --json runs list --project harness-evolver-{best_version} --failed --fields id,name,error,inputs --limit 10 > .harness-evolver/langsmith_diagnosis.json 2>/dev/null || echo "[]" > .harness-evolver/langsmith_diagnosis.json
+langsmith-cli --json runs list --project harness-evolver-{last_evaluated_version} --failed --fields id,name,error,inputs --limit 10 > .harness-evolver/langsmith_diagnosis.json 2>/dev/null || echo "[]" > .harness-evolver/langsmith_diagnosis.json
 
-langsmith-cli --json runs stats --project harness-evolver-{best_version} > .harness-evolver/langsmith_stats.json 2>/dev/null || echo "{}" > .harness-evolver/langsmith_stats.json
+langsmith-cli --json runs stats --project harness-evolver-{last_evaluated_version} > .harness-evolver/langsmith_stats.json 2>/dev/null || echo "{}" > .harness-evolver/langsmith_stats.json
 ```
 
-**Context7 (if available):**
+For the first iteration, use `baseline` as the version. For subsequent iterations, use the latest evaluated version.
 
-Check `config.json` field `stack.detected`. For each detected library, use the Context7 MCP tools to fetch relevant documentation:
-
-```
-For each library in stack.detected:
-  1. resolve-library-id with the context7_id
-  2. get-library-docs with a query relevant to the current failure modes
-  3. Save output to .harness-evolver/context7_docs.md (append each library's docs)
-```
-
-This runs ONCE per iteration, not per library. Focus on the library most relevant to the current failures.
-
-If Context7 MCP is not available, skip silently.
+These files are included in the proposer's `<files_to_read>` so it has real trace data for diagnosis.
 
 ### 2. Propose
 
-Dispatch a subagent using the **Agent tool** with `subagent_type: "harness-evolver-proposer"`.
+Dispatch a subagent using the **Agent tool**.
 
-The prompt MUST be the full XML block below. The `<files_to_read>` MUST include the LangSmith/Context7 files if they were gathered in step 1.5.
+First, read the proposer agent definition to include in the prompt:
+```bash
+cat ~/.claude/agents/harness-evolver-proposer.md
+```
+
+Then dispatch the Agent with the agent definition + structured task:
 
 ```
 Agent(
-  subagent_type: "harness-evolver-proposer",
   description: "Propose harness {version}",
   prompt: |
+    <agent_instructions>
+    {paste the FULL content of harness-evolver-proposer.md here}
+    </agent_instructions>
+
     <objective>
     Propose harness version {version} that improves on the current best score of {best_score}.
     </objective>
@@ -93,7 +81,6 @@ Agent(
     - .harness-evolver/harnesses/{best_version}/proposal.md
     - .harness-evolver/langsmith_diagnosis.json (if exists — LangSmith failure analysis)
     - .harness-evolver/langsmith_stats.json (if exists — LangSmith aggregate stats)
-    - .harness-evolver/context7_docs.md (if exists — current library documentation)
     - .harness-evolver/architecture.json (if exists — architect topology recommendation)
     </files_to_read>
 
@@ -107,13 +94,13 @@ Agent(
     <success_criteria>
     - harness.py maintains CLI interface (--input, --output, --traces-dir, --config)
     - proposal.md documents evidence-based reasoning
-    - Changes are motivated by trace analysis (LangSmith data if available), not guesswork
-    - If context7_docs.md was provided, API usage must match current documentation
+    - If proposing API changes, MUST use Context7 (resolve-library-id + get-library-docs) to verify current docs
+    - Changes motivated by LangSmith trace data (in langsmith_diagnosis.json) when available
     </success_criteria>
 )
 ```
 
-Wait for `## PROPOSAL COMPLETE` in the response. The subagent gets a fresh context window and loads the agent definition from `~/.claude/agents/harness-evolver-proposer.md`.
+Wait for `## PROPOSAL COMPLETE` in the response.
 
 ### 3. Validate
 
@@ -170,13 +157,21 @@ python3 $TOOLS/evaluate.py run \
     --timeout 60
 ```
 
-Dispatch critic subagent using the **Agent tool** with `subagent_type: "harness-evolver-critic"`:
+First read the critic agent definition:
+```bash
+cat ~/.claude/agents/harness-evolver-critic.md
+```
+
+Then dispatch:
 
 ```
 Agent(
-  subagent_type: "harness-evolver-critic",
   description: "Critic: analyze eval quality",
   prompt: |
+    <agent_instructions>
+    {paste the FULL content of harness-evolver-critic.md here}
+    </agent_instructions>
+
     <objective>
     EVAL GAMING DETECTED: Score jumped from {parent_score} to {score} in one iteration.
     Analyze the eval quality and propose a stricter eval.
@@ -239,13 +234,21 @@ python3 $TOOLS/analyze_architecture.py \
     -o .harness-evolver/architecture_signals.json
 ```
 
-Dispatch architect subagent using the **Agent tool** with `subagent_type: "harness-evolver-architect"`:
+First read the architect agent definition:
+```bash
+cat ~/.claude/agents/harness-evolver-architect.md
+```
+
+Then dispatch:
 
 ```
 Agent(
-  subagent_type: "harness-evolver-architect",
   description: "Architect: analyze topology after {stagnation/regression}",
   prompt: |
+    <agent_instructions>
+    {paste the FULL content of harness-evolver-architect.md here}
+    </agent_instructions>
+
     <objective>
     The evolution loop has {stagnated/regressed} after {iterations} iterations (best: {best_score}).
     Analyze the harness architecture and recommend a topology change.

package/tools/init.py

@@ -298,10 +298,26 @@ def main():
             print("  Recommendation: install langsmith-cli for rich trace analysis:")
             print("    uv tool install langsmith-cli && langsmith-cli auth login")
 
-    # Detect stack
-    stack = _detect_stack(args.harness)
+    # Detect stack — try original harness first, then baseline copy, then scan entire source dir
+    stack = _detect_stack(os.path.abspath(args.harness))
+    if not stack:
+        stack = _detect_stack(os.path.join(base, "baseline", "harness.py"))
+    if not stack:
+        # Scan the original directory for any .py files with known imports
+        harness_dir = os.path.dirname(os.path.abspath(args.harness))
+        detect_stack_py = os.path.join(os.path.dirname(__file__), "detect_stack.py")
+        if os.path.exists(detect_stack_py):
+            try:
+                r = subprocess.run(
+                    ["python3", detect_stack_py, harness_dir],
+                    capture_output=True, text=True, timeout=30,
+                )
+                if r.returncode == 0 and r.stdout.strip():
+                    stack = json.loads(r.stdout)
+            except Exception:
+                pass
     config["stack"] = {
-        "detected": stack,
+        "detected": stack if stack else {},
         "documentation_hint": "use context7",
         "auto_detected": True,
     }