harness-evolver 2.1.0 → 2.3.0

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "harness-evolver",
-  "version": "2.1.0",
+  "version": "2.3.0",
   "description": "Meta-Harness-style autonomous harness optimization for Claude Code",
   "author": "Raphael Valdetaro",
   "license": "MIT",

package/skills/architect/SKILL.md

@@ -48,21 +48,13 @@ python3 $TOOLS/analyze_architecture.py \
     -o .harness-evolver/architecture_signals.json
 ```
 
-3. Read the architect agent definition:
-```bash
-cat ~/.claude/agents/harness-evolver-architect.md
-```
-
-4. Dispatch using the Agent tool — include the agent definition in the prompt:
+3. Dispatch using the Agent tool with `subagent_type: "harness-evolver-architect"`:
 
 ```
 Agent(
+  subagent_type: "harness-evolver-architect",
   description: "Architect: topology analysis",
   prompt: |
-    <agent_instructions>
-    {paste the FULL content of harness-evolver-architect.md here}
-    </agent_instructions>
-
     <objective>
     Analyze the harness architecture and recommend the optimal multi-agent topology.
     {If called from evolve: "The evolution loop stagnated/regressed after N iterations."}

package/skills/critic/SKILL.md

@@ -22,21 +22,13 @@ TOOLS=$([ -d ".harness-evolver/tools" ] && echo ".harness-evolver/tools" || echo
 
 1. Read `summary.json` and identify the suspicious pattern (score jump, premature convergence).
 
-2. Read the critic agent definition:
-```bash
-cat ~/.claude/agents/harness-evolver-critic.md
-```
-
-3. Dispatch using the Agent tool — include the agent definition in the prompt:
+2. Dispatch using the Agent tool with `subagent_type: "harness-evolver-critic"`:
 
 ```
 Agent(
+  subagent_type: "harness-evolver-critic",
   description: "Critic: analyze eval quality",
   prompt: |
-    <agent_instructions>
-    {paste the FULL content of harness-evolver-critic.md here}
-    </agent_instructions>
-
     <objective>
     Analyze eval quality for this harness evolution project.
     The best version is {version} with score {score} achieved in {iterations} iteration(s).

package/skills/evolve/SKILL.md

@@ -52,49 +52,105 @@ LS_PROJECT=$(langsmith-cli --json projects list --name-pattern "harness-evolver*
 
 If `LS_PROJECT` is empty, langsmith-cli is not available or no projects exist — skip to step 2.
 
-**Step 2: Gather traces from the discovered project**
+**Step 2: Gather raw traces from the discovered project**
 
 ```bash
 if [ -n "$LS_PROJECT" ]; then
-  langsmith-cli --json runs list --project "$LS_PROJECT" --failed --fields id,name,error,inputs --limit 10 > .harness-evolver/langsmith_diagnosis.json 2>/dev/null || echo "[]" > .harness-evolver/langsmith_diagnosis.json
-  langsmith-cli --json runs list --project "$LS_PROJECT" --fields id,name,inputs,outputs,latency_ms,total_tokens --limit 20 > .harness-evolver/langsmith_runs.json 2>/dev/null || echo "[]" > .harness-evolver/langsmith_runs.json
+  langsmith-cli --json runs list --project "$LS_PROJECT" --recent --fields id,name,inputs,outputs,error,total_tokens --limit 30 > /tmp/langsmith_raw.json 2>/dev/null || echo "[]" > /tmp/langsmith_raw.json
   langsmith-cli --json runs stats --project "$LS_PROJECT" > .harness-evolver/langsmith_stats.json 2>/dev/null || echo "{}" > .harness-evolver/langsmith_stats.json
   echo "$LS_PROJECT" > .harness-evolver/langsmith_project.txt
 else
-  echo "[]" > .harness-evolver/langsmith_diagnosis.json
+  echo "[]" > /tmp/langsmith_raw.json
   echo "{}" > .harness-evolver/langsmith_stats.json
 fi
 ```
 
-These files are included in the proposer's `<files_to_read>` so it has real trace data for diagnosis.
+**Step 3: Process raw LangSmith data into a readable format for proposers**
+
+The raw langsmith data has LangChain-serialized messages that are hard to read. Process it into a clean summary:
+
+```bash
+python3 -c "
+import json, sys
+
+raw = json.load(open('/tmp/langsmith_raw.json'))
+if not raw:
+    json.dump([], open('.harness-evolver/langsmith_runs.json', 'w'))
+    sys.exit(0)
+
+clean = []
+for r in raw:
+    entry = {'name': r.get('name', '?'), 'tokens': r.get('total_tokens', 0), 'error': r.get('error')}
+
+    # Extract readable prompt from LangChain serialized inputs
+    inputs = r.get('inputs', {})
+    if isinstance(inputs, dict) and 'messages' in inputs:
+        msgs = inputs['messages']
+        for msg_group in (msgs if isinstance(msgs, list) else [msgs]):
+            for msg in (msg_group if isinstance(msg_group, list) else [msg_group]):
+                if isinstance(msg, dict):
+                    kwargs = msg.get('kwargs', msg)
+                    content = kwargs.get('content', '')
+                    msg_type = msg.get('id', ['','','',''])[3] if isinstance(msg.get('id'), list) else 'unknown'
+                    if 'Human' in str(msg_type) or 'user' in str(msg_type).lower():
+                        entry['user_message'] = str(content)[:300]
+                    elif 'System' in str(msg_type):
+                        entry['system_prompt_preview'] = str(content)[:200]
+
+    # Extract readable output
+    outputs = r.get('outputs', {})
+    if isinstance(outputs, dict) and 'generations' in outputs:
+        gens = outputs['generations']
+        if gens and isinstance(gens, list) and gens[0]:
+            gen = gens[0][0] if isinstance(gens[0], list) else gens[0]
+            if isinstance(gen, dict):
+                msg = gen.get('message', gen)
+                if isinstance(msg, dict):
+                    kwargs = msg.get('kwargs', msg)
+                    entry['llm_response'] = str(kwargs.get('content', ''))[:300]
+
+    clean.append(entry)
+
+json.dump(clean, open('.harness-evolver/langsmith_runs.json', 'w'), indent=2, ensure_ascii=False)
+print(f'Processed {len(clean)} LangSmith runs into readable format')
+" 2>/dev/null || echo "[]" > .harness-evolver/langsmith_runs.json
+```
+
+The resulting `langsmith_runs.json` has clean, readable entries:
+```json
+[
+  {
+    "name": "ChatGoogleGenerativeAI",
+    "tokens": 1332,
+    "error": null,
+    "user_message": "Analise este texto: Bom dia pessoal...",
+    "system_prompt_preview": "Você é um moderador de conteúdo...",
+    "llm_response": "{\"categories\": [\"safe\"], \"severity\": \"safe\"...}"
+  }
+]
+```
+
+These files are included in the proposer's `<files_to_read>` so it has readable trace data for diagnosis.
 
 ### 2. Propose (3 parallel candidates)
 
 Spawn 3 proposer agents IN PARALLEL, each with a different evolutionary strategy.
 This follows the DGM/AlphaEvolve pattern: exploit + explore + crossover.
 
-First, read the proposer agent definition:
-```bash
-cat ~/.claude/agents/harness-evolver-proposer.md
-```
-
-Then determine parents for each strategy:
+Determine parents for each strategy:
 - **Exploiter parent**: current best version (from summary.json `best.version`)
 - **Explorer parent**: a non-best version with low offspring count (read summary.json history, pick one that scored >0 but is NOT the best and has NOT been parent to many children)
 - **Crossover parents**: best version + a different high-scorer from a different lineage
 
-Spawn all 3 using the Agent tool. The first 2 use `run_in_background: true`, the 3rd blocks:
+Spawn all 3 using the Agent tool with `subagent_type: "harness-evolver-proposer"`. The first 2 use `run_in_background: true`, the 3rd blocks:
 
 **Candidate A (Exploiter)** — `run_in_background: true`:
 ```
 Agent(
+  subagent_type: "harness-evolver-proposer",
   description: "Proposer A (exploit): targeted fix for {version}",
   run_in_background: true,
   prompt: |
-    <agent_instructions>
-    {FULL content of harness-evolver-proposer.md}
-    </agent_instructions>
-
     <strategy>
     APPROACH: exploitation
     You are the EXPLOITER. Make the SMALLEST, most targeted change that fixes
@@ -130,13 +186,10 @@ Agent(
 **Candidate B (Explorer)** — `run_in_background: true`:
 ```
 Agent(
+  subagent_type: "harness-evolver-proposer",
   description: "Proposer B (explore): bold change from {explorer_parent}",
   run_in_background: true,
   prompt: |
-    <agent_instructions>
-    {FULL content of harness-evolver-proposer.md}
-    </agent_instructions>
-
     <strategy>
     APPROACH: exploration
     You are the EXPLORER. Try a FUNDAMENTALLY DIFFERENT approach.
@@ -172,12 +225,9 @@ Agent(
 **Candidate C (Crossover)** — blocks (last one):
 ```
 Agent(
+  subagent_type: "harness-evolver-proposer",
   description: "Proposer C (crossover): combine {parent_a} + {parent_b}",
   prompt: |
-    <agent_instructions>
-    {FULL content of harness-evolver-proposer.md}
-    </agent_instructions>
-
     <strategy>
     APPROACH: crossover
     You are the CROSSOVER agent. Combine the STRENGTHS of two different versions:
@@ -269,21 +319,13 @@ python3 $TOOLS/evaluate.py run \
 
 For each evaluated candidate, read its scores.json. If `eval_type` is `"pending-judge"` (combined_score == -1), the eval was a passthrough and needs judge scoring.
 
-Read the judge agent definition:
-```bash
-cat ~/.claude/agents/harness-evolver-judge.md
-```
-
-Spawn judge subagent for EACH candidate that needs judging:
+Spawn judge subagent with `subagent_type: "harness-evolver-judge"` for EACH candidate that needs judging:
 
 ```
 Agent(
+  subagent_type: "harness-evolver-judge",
   description: "Judge: score {version}{suffix} outputs",
   prompt: |
-    <agent_instructions>
-    {FULL content of harness-evolver-judge.md}
-    </agent_instructions>
-
     <objective>
     Score the outputs of harness version {version}{suffix} across all {N} tasks.
     </objective>
@@ -354,21 +396,13 @@ python3 $TOOLS/evaluate.py run \
     --timeout 60
 ```
 
-First read the critic agent definition:
-```bash
-cat ~/.claude/agents/harness-evolver-critic.md
-```
-
-Then dispatch:
+Dispatch the critic agent:
 
 ```
 Agent(
+  subagent_type: "harness-evolver-critic",
   description: "Critic: analyze eval quality",
   prompt: |
-    <agent_instructions>
-    {paste the FULL content of harness-evolver-critic.md here}
-    </agent_instructions>
-
     <objective>
     EVAL GAMING DETECTED: Score jumped from {parent_score} to {score} in one iteration.
     Analyze the eval quality and propose a stricter eval.
@@ -431,21 +465,13 @@ python3 $TOOLS/analyze_architecture.py \
     -o .harness-evolver/architecture_signals.json
 ```
 
-First read the architect agent definition:
-```bash
-cat ~/.claude/agents/harness-evolver-architect.md
-```
-
-Then dispatch:
+Dispatch the architect agent:
 
 ```
 Agent(
+  subagent_type: "harness-evolver-architect",
   description: "Architect: analyze topology after {stagnation/regression}",
   prompt: |
-    <agent_instructions>
-    {paste the FULL content of harness-evolver-architect.md here}
-    </agent_instructions>
-
     <objective>
     The evolution loop has {stagnated/regressed} after {iterations} iterations (best: {best_score}).
     Analyze the harness architecture and recommend a topology change.

package/skills/init/SKILL.md

@@ -49,19 +49,12 @@ If NO eval exists:
 **Tasks** (`tasks/`): If test tasks exist, use them.
 
 If NO tasks exist:
-- Read the testgen agent definition:
-  ```bash
-  cat ~/.claude/agents/harness-evolver-testgen.md
-  ```
-- Spawn testgen subagent:
+- Spawn testgen subagent with `subagent_type: "harness-evolver-testgen"`:
   ```
   Agent(
+    subagent_type: "harness-evolver-testgen",
     description: "TestGen: generate test cases for this project",
     prompt: |
-      <agent_instructions>
-      {FULL content of harness-evolver-testgen.md}
-      </agent_instructions>
-
       <objective>
       Generate 30 diverse test cases for this project. Write them to tasks/ directory.
       </objective>