harness-evolver 2.5.1 → 2.6.0

package/README.md

@@ -46,12 +46,20 @@ claude
 
 <table>
 <tr>
-<td><b>5 Proposers</b></td>
-<td>Each iteration spawns 5 parallel agents with different strategies: exploit (targeted fix), explore (bold rewrite), crossover (combine two parents), prompt specialist, retrieval specialist. Best candidate wins.</td>
+<td><b>5 Adaptive Proposers</b></td>
+<td>Each iteration spawns 5 parallel agents: exploit (targeted fix), explore (bold rewrite), crossover (combine two parents), and 2 failure-focused agents that target the weakest task clusters. Strategies adapt every iteration based on actual per-task scores — no fixed specialists.</td>
 </tr>
 <tr>
-<td><b>Full Traces</b></td>
-<td>Every harness run captures stdout, stderr, timing, and per-task I/O. LangSmith auto-tracing for LangChain/LangGraph agents. Proposers read actual LLM prompts and responses.</td>
+<td><b>Trace Insights</b></td>
+<td>Every harness run captures stdout, stderr, timing, and per-task I/O. LangSmith auto-tracing for LangChain/LangGraph agents. Traces are systematically clustered by error pattern, token usage, and response type — proposers receive structured diagnostic data, not raw logs.</td>
+</tr>
+<tr>
+<td><b>Quality-Diversity Selection</b></td>
+<td>Not winner-take-all. Tracks per-task champions — a candidate that loses overall but excels at specific tasks is preserved as the next crossover parent. The archive never discards variants.</td>
+</tr>
+<tr>
+<td><b>Durable Test Gates</b></td>
+<td>When the loop fixes a failure, regression tasks are automatically generated to lock in the improvement. The test suite grows over iterations — fixed bugs can never silently return.</td>
 </tr>
 <tr>
 <td><b>Critic</b></td>
@@ -74,13 +82,14 @@ claude
 | Command | What it does |
 |---|---|
 | `/harness-evolver:init` | Scan project, create harness/eval/tasks, run baseline |
-| `/harness-evolver:evolve` | Run the autonomous optimization loop (5 parallel proposers) |
+| `/harness-evolver:evolve` | Run the autonomous optimization loop (5 adaptive proposers) |
 | `/harness-evolver:status` | Show progress, scores, stagnation detection |
 | `/harness-evolver:compare` | Diff two versions with per-task analysis |
 | `/harness-evolver:diagnose` | Deep trace analysis of a specific version |
 | `/harness-evolver:deploy` | Promote the best harness back to your project |
 | `/harness-evolver:architect` | Analyze and recommend optimal agent topology |
 | `/harness-evolver:critic` | Evaluate eval quality and detect gaming |
+| `/harness-evolver:import-traces` | Pull production LangSmith traces as eval tasks |
 
 ---
 
@@ -139,16 +148,20 @@ Works with any language, any framework, any domain. If your project doesn't have
 ```
 /harness-evolver:evolve
   │
-  ├─ 1. Gather LangSmith traces (processed into readable format)
-  ├─ 2. Spawn 5 proposers in parallel (exploit/explore/crossover/prompt/retrieval)
-  ├─ 3. Validate all candidates
-  ├─ 4. Evaluate all candidates
+  ├─ 1.  Get next version
+  ├─ 1.5 Gather LangSmith traces (processed into readable format)
+  ├─ 1.6 Generate Trace Insights (cluster errors, analyze tokens, cross-ref scores)
+  ├─ 1.8 Analyze per-task failures (cluster by category for adaptive briefings)
+  ├─ 2.  Spawn 5 proposers in parallel (exploit / explore / crossover / 2× failure-targeted)
+  ├─ 3.  Validate all candidates
+  ├─ 4.  Evaluate all candidates
   ├─ 4.5 Judge (if using LLM-as-judge eval)
-  ├─ 5. Select winner (highest combined_score)
-  ├─ 6. Report results
+  ├─ 5.  Select winner + track per-task champion
+  ├─ 5.5 Test suite growth (generate regression tasks for fixed failures)
+  ├─ 6.  Report results
   ├─ 6.5 Auto-trigger Critic (if score jumped >0.3 or reached 1.0 too fast)
-  ├─ 7. Auto-trigger Architect (if regression or stagnation)
-  └─ 8. Check stop conditions (target reached, N iterations, stagnation post-architect)
+  ├─ 7.  Auto-trigger Architect (if regression or stagnation)
+  └─ 8.  Check stop conditions (target reached, N iterations, stagnation post-architect)
 ```
 
 ---
@@ -175,9 +188,12 @@ The plugin auto-detects available keys. No key needed for the included example.
 |---|---|---|---|---|
 | **Format** | Paper artifact | Framework (Docker) | Plugin (passive) | **Plugin (active)** |
 | **Search** | Code-space | Code-space | Prompt-space | **Code-space** |
-| **Candidates/iter** | 1 | 1 | N/A | **5 parallel** |
+| **Candidates/iter** | 1 | 1 | N/A | **5 parallel (adaptive)** |
+| **Selection** | Single best | Single best | N/A | **Quality-diversity (per-task)** |
 | **Auto-critique** | No | No | No | **Yes (critic + judge)** |
 | **Architecture** | Fixed | Fixed | N/A | **Auto-recommended** |
+| **Trace analysis** | Manual | No | No | **Systematic (clustering + insights)** |
+| **Test growth** | No | No | No | **Yes (durable regression gates)** |
 | **LangSmith** | No | No | No | **Yes** |
 | **Context7** | No | No | No | **Yes** |
 | **Zero-config** | No | No | No | **Yes** |

package/agents/harness-evolver-proposer.md

@@ -26,6 +26,20 @@ Your prompt contains a `<strategy>` block defining your approach. Follow it:
 
 If no strategy block is present, default to exploitation (conservative improvement).
 
+## Trace Insights
+
+If `.harness-evolver/trace_insights.json` exists in your `<files_to_read>`, use it to guide your diagnosis:
+
+1. Check `top_issues` first — these are the highest-impact problems sorted by severity
+2. Check `hypotheses` for data-driven theories about failure causes
+3. Use `error_clusters` to understand which error patterns affect which runs
+4. The `token_analysis` and `token_score_correlation` sections show if verbosity correlates with quality
+5. `score_cross_ref.failure_categories` maps failure patterns to task categories
+
+These insights are generated from LangSmith traces cross-referenced with per-task scores — they are **data, not guesses**. Prioritize addressing issues marked severity `"high"` over `"medium"` or `"low"`.
+
+If trace insights are not available, proceed with manual trace analysis as described in Phase 2.
+
 ## Context7 — Enrich Your Knowledge
 
 You have access to Context7 MCP tools (`resolve-library-id` and `get-library-docs`) for looking up **current, version-specific documentation** of any library.

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "harness-evolver",
-  "version": "2.5.1",
+  "version": "2.6.0",
   "description": "Meta-Harness-style autonomous harness optimization for Claude Code",
   "author": "Raphael Valdetaro",
   "license": "MIT",

package/skills/evolve/SKILL.md

@@ -132,6 +132,32 @@ The resulting `langsmith_runs.json` has clean, readable entries:
 
 These files are included in the proposer's `<files_to_read>` so it has readable trace data for diagnosis.
 
+### 1.6. Generate Trace Insights (systematic analysis)
+
+If LangSmith traces were gathered, run systematic analysis to cluster errors, analyze token usage, and cross-reference with scores:
+
+```bash
+if [ -f ".harness-evolver/langsmith_runs.json" ]; then
+    BEST=$(python3 -c "import json; s=json.load(open('.harness-evolver/summary.json')); print(s['best']['version'])")
+    SCORES_PATH=".harness-evolver/harnesses/$BEST/scores.json"
+    [ ! -f "$SCORES_PATH" ] && SCORES_PATH=".harness-evolver/baseline/scores.json"
+    python3 $TOOLS/trace_insights.py \
+        --langsmith-runs .harness-evolver/langsmith_runs.json \
+        --langsmith-stats .harness-evolver/langsmith_stats.json \
+        --scores "$SCORES_PATH" \
+        --tasks-dir .harness-evolver/eval/tasks/ \
+        --output .harness-evolver/trace_insights.json 2>/dev/null
+fi
+```
+
+The resulting `trace_insights.json` contains:
+- `error_clusters`: grouped error patterns with counts
+- `token_analysis`: score distribution by token usage bucket (low/medium/high)
+- `hypotheses`: data-driven theories about failure causes
+- `top_issues`: highest-impact problems sorted by severity
+
+This file is included in all proposers' `<files_to_read>` so they have structured diagnostic data.
+
 ### 1.8. Analyze Per-Task Failures (adaptive briefings for Candidates D and E)
 
 Before spawning proposers, analyze which tasks are failing and cluster them:
@@ -228,6 +254,7 @@ Agent(
     - .harness-evolver/langsmith_diagnosis.json (if exists)
     - .harness-evolver/langsmith_stats.json (if exists)
     - .harness-evolver/langsmith_runs.json (if exists)
+    - .harness-evolver/trace_insights.json (if exists)
     - .harness-evolver/architecture.json (if exists)
     </files_to_read>
 
@@ -267,6 +294,7 @@ Agent(
     - .harness-evolver/harnesses/{explorer_parent}/scores.json
     - .harness-evolver/langsmith_diagnosis.json (if exists)
     - .harness-evolver/langsmith_runs.json (if exists)
+    - .harness-evolver/trace_insights.json (if exists)
     - .harness-evolver/architecture.json (if exists)
     </files_to_read>
 
@@ -305,6 +333,7 @@ Agent(
     - .harness-evolver/harnesses/{parent_b}/scores.json
     - .harness-evolver/langsmith_diagnosis.json (if exists)
     - .harness-evolver/langsmith_runs.json (if exists)
+    - .harness-evolver/trace_insights.json (if exists)
     - .harness-evolver/architecture.json (if exists)
     </files_to_read>
 
@@ -347,6 +376,7 @@ Agent(
     - .harness-evolver/harnesses/{best_version}/harness.py
     - .harness-evolver/harnesses/{best_version}/scores.json
     - .harness-evolver/langsmith_runs.json (if exists)
+    - .harness-evolver/trace_insights.json (if exists)
     - .harness-evolver/architecture.json (if exists)
     </files_to_read>
 
@@ -407,6 +437,7 @@ Agent(
     - .harness-evolver/harnesses/{best_version}/harness.py
     - .harness-evolver/harnesses/{best_version}/scores.json
     - .harness-evolver/langsmith_runs.json (if exists)
+    - .harness-evolver/trace_insights.json (if exists)
     - .harness-evolver/architecture.json (if exists)
     </files_to_read>
 
@@ -580,6 +611,34 @@ Iteration {i}/{N} — {num_candidates} candidates evaluated:
 
 Keep losing candidates in their directories (they're part of the archive — never discard, per DGM).
 
+### 5.5. Test Suite Growth (Durable Regression Gates)
+
+After the winner is promoted, check if any previously-failing tasks are now passing.
+Generate regression tasks to lock in improvements and prevent future regressions:
+
+```bash
+PREV_BEST=$(python3 -c "
+import json
+s = json.load(open('.harness-evolver/summary.json'))
+versions = s.get('versions', [])
+print(versions[-2]['version'] if len(versions) >= 2 else '')
+" 2>/dev/null)
+if [ -n "$PREV_BEST" ] && [ -f ".harness-evolver/harnesses/$PREV_BEST/scores.json" ]; then
+    python3 $TOOLS/test_growth.py \
+        --current-scores .harness-evolver/harnesses/{version}/scores.json \
+        --previous-scores ".harness-evolver/harnesses/$PREV_BEST/scores.json" \
+        --tasks-dir .harness-evolver/eval/tasks/ \
+        --output-dir .harness-evolver/eval/tasks/ \
+        --max-total-tasks 60 2>/dev/null
+fi
+```
+
+If new tasks were added, print: "Added {N} regression tasks to lock in improvements on: {task_ids}"
+
+This is the "durable test gates" pattern: every fixed failure becomes a permanent regression test.
+New tasks are tagged with `metadata.type: "regression"` and `metadata.source: "regression"` so they
+can be distinguished from original tasks. The test suite only grows — regression tasks are never removed.
+
 ### 6. Report
 
 Read `summary.json`. Print: `Iteration {i}/{N}: {version} scored {score} (best: {best} at {best_score})`

package/skills/import-traces/SKILL.md

@@ -0,0 +1,102 @@
+---
+name: harness-evolver:import-traces
+description: "Use when the user wants to import real production traces from LangSmith as test tasks, convert traces to eval tasks, enrich their eval set with real-world data, or pull production data into their harness evaluation."
+argument-hint: "[--project NAME] [--limit N]"
+allowed-tools: [Read, Write, Edit, Bash, Glob, Grep]
+---
+
+# /harness-evolver:import-traces
+
+Import production traces from LangSmith and convert them into eval tasks. This enriches the test suite with real-world inputs, prioritizing traces with negative user feedback.
+
+## Prerequisites
+
+- `.harness-evolver/` must exist. If not, tell user to run `harness-evolver:init` first.
+- `langsmith-cli` must be available. Check:
+
+```bash
+which langsmith-cli 2>/dev/null
+```
+
+If not found: "Install langsmith-cli first: `uv tool install langsmith-cli && langsmith-cli auth login`"
+
+## Resolve Tool Path
+
+```bash
+TOOLS=$([ -d ".harness-evolver/tools" ] && echo ".harness-evolver/tools" || echo "$HOME/.harness-evolver/tools")
+```
+
+## Parse Arguments
+
+- `--project NAME` — LangSmith project name (if not provided, discover interactively)
+- `--limit N` — max traces to import (default: 20)
+
+## Phase 1: Discover Projects
+
+If `--project` not provided, list available projects:
+
+```bash
+langsmith-cli --json projects list --limit 20 2>/dev/null
+```
+
+Show the user a list of projects with run counts. Let them pick one, or use the most recent.
+
+If `--project` is provided, use it directly.
+
+## Phase 2: Fetch Traces
+
+```bash
+langsmith-cli --json runs list \
+    --project "{project_name}" \
+    --limit {limit} \
+    --fields id,name,inputs,outputs,error,feedback_stats,total_tokens \
+    > /tmp/harness_import_traces.json 2>/dev/null
+```
+
+Check the output has data:
+```bash
+python3 -c "import json; data=json.load(open('/tmp/harness_import_traces.json')); print(f'{len(data)} traces fetched')"
+```
+
+If no traces found, tell user the project may be empty or the name may be wrong.
+
+## Phase 3: Convert to Tasks
+
+```bash
+python3 $TOOLS/import_traces.py \
+    --traces-json /tmp/harness_import_traces.json \
+    --output-dir .harness-evolver/eval/tasks/ \
+    --prefix imported \
+    --max-tasks {limit}
+```
+
+## Phase 4: Report
+
+Read the tool output and report:
+- How many traces were imported
+- How many had negative feedback (high priority)
+- How many were skipped (no extractable input, duplicates)
+- Total tasks now in eval set
+
+```bash
+ls .harness-evolver/eval/tasks/*.json | wc -l
+```
+
+Print:
+```
+Imported {N} production traces as eval tasks.
+  {M} with negative user feedback (high priority)
+  {K} skipped (no input or duplicates)
+  Total eval tasks: {total}
+
+Next: run `harness-evolver:evolve` to optimize against real-world inputs.
+```
+
+## Gotchas
+
+- Traces with no extractable user input are skipped (e.g., system-only runs)
+- Duplicate traces (same run ID) are automatically skipped
+- Imported tasks are tagged with `metadata.source: "imported"` and `metadata.type: "production"`
+- Tasks with negative feedback get `metadata.user_feedback: "negative"` — the proposer should prioritize these
+- The `metadata.langsmith_run_id` field links back to the original trace for debugging
+- Cleanup: `rm /tmp/harness_import_traces.json` after import

package/tools/import_traces.py

@@ -0,0 +1,229 @@
+#!/usr/bin/env python3
+"""Import LangSmith Traces as Eval Tasks for Harness Evolver.
+
+Transforms LangSmith trace JSON (from langsmith-cli) into task JSON files
+for the evaluation set. Prioritizes traces with negative feedback.
+
+Usage:
+    python3 import_traces.py \
+        --traces-json /tmp/langsmith_traces.json \
+        --output-dir .harness-evolver/eval/tasks/ \
+        --prefix imported \
+        [--max-tasks 30]
+
+Stdlib-only. No external dependencies.
+"""
+
+import argparse
+import hashlib
+import json
+import os
+import re
+import sys
+
+
+def load_json(path):
+    """Load JSON file, return None if missing or invalid."""
+    if not path or not os.path.exists(path):
+        return None
+    try:
+        with open(path) as f:
+            return json.load(f)
+    except (json.JSONDecodeError, OSError):
+        return None
+
+
+def extract_input_from_trace(run):
+    """Extract the user input from a LangSmith run's inputs field.
+
+    Handles multiple LangChain serialization formats:
+    - Direct {"input": "..."} field
+    - {"messages": [[HumanMessage, ...]]} format
+    - {"question": "..."} or {"query": "..."} fields
+    """
+    inputs = run.get("inputs", {})
+    if not inputs:
+        return None
+
+    if isinstance(inputs, str):
+        return inputs
+
+    # Direct input field
+    for key in ("input", "question", "query", "prompt", "text", "user_input"):
+        if key in inputs and isinstance(inputs[key], str):
+            return inputs[key]
+
+    # LangChain messages format
+    messages = inputs.get("messages") or inputs.get("input")
+    if isinstance(messages, list):
+        # Might be [[msg1, msg2]] (batched) or [msg1, msg2]
+        if messages and isinstance(messages[0], list):
+            messages = messages[0]
+        for msg in messages:
+            if isinstance(msg, dict):
+                # {"type": "human", "content": "..."}
+                if msg.get("type") in ("human", "HumanMessage") or msg.get("role") == "user":
+                    content = msg.get("content", "")
+                    if isinstance(content, str) and content:
+                        return content
+                    if isinstance(content, list):
+                        # Multi-modal: [{"type": "text", "text": "..."}]
+                        for part in content:
+                            if isinstance(part, dict) and part.get("type") == "text":
+                                return part.get("text", "")
+            elif isinstance(msg, str) and msg:
+                return msg
+
+    # Fallback: stringify the whole inputs
+    flat = json.dumps(inputs)
+    if len(flat) > 20:  # Only if there's meaningful content
+        return flat[:2000]
+
+    return None
+
+
+def extract_feedback(run):
+    """Extract user feedback from a LangSmith run."""
+    feedback = run.get("feedback_stats") or run.get("feedback") or {}
+    if not feedback:
+        return None
+
+    # feedback_stats format: {"thumbs_up": N, "thumbs_down": N}
+    if isinstance(feedback, dict):
+        up = feedback.get("thumbs_up", 0) or feedback.get("positive", 0)
+        down = feedback.get("thumbs_down", 0) or feedback.get("negative", 0)
+        if down > 0:
+            return "negative"
+        if up > 0:
+            return "positive"
+    return None
+
+
+def infer_difficulty(text):
+    """Infer difficulty from input characteristics."""
+    if not text:
+        return "medium"
+    length = len(text)
+    # Count question marks, clauses, etc.
+    questions = text.count("?")
+    sentences = len(re.split(r"[.!?]+", text))
+
+    if length < 50 and questions <= 1:
+        return "easy"
+    if length > 500 or questions > 2 or sentences > 5:
+        return "hard"
+    return "medium"
+
+
+def short_id(run_id):
+    """Create a short deterministic ID from a full run ID."""
+    return hashlib.md5(str(run_id).encode()).hexdigest()[:8]
+
+
+def main():
+    parser = argparse.ArgumentParser(description="Import LangSmith traces as eval tasks")
+    parser.add_argument("--traces-json", required=True, help="Path to langsmith-cli JSON output")
+    parser.add_argument("--output-dir", required=True, help="Directory to write task JSON files")
+    parser.add_argument("--prefix", default="imported", help="Prefix for task IDs (default: imported)")
+    parser.add_argument("--max-tasks", type=int, default=30, help="Max tasks to import (default: 30)")
+    parser.add_argument("--prioritize-negative", action="store_true", default=True,
+                        help="Import negative-feedback traces first (default: true)")
+    args = parser.parse_args()
+
+    traces = load_json(args.traces_json)
+    if not traces:
+        print("No traces found or invalid JSON — nothing to import")
+        return
+
+    if isinstance(traces, dict):
+        # Might be wrapped in {"runs": [...]}
+        traces = traces.get("runs", traces.get("data", [traces]))
+
+    if not isinstance(traces, list):
+        print("Unexpected traces format — expected a JSON array")
+        return
+
+    # Sort: negative feedback first, then errors, then the rest
+    if args.prioritize_negative:
+        def priority(run):
+            fb = extract_feedback(run)
+            has_error = bool(run.get("error"))
+            if fb == "negative":
+                return 0
+            if has_error:
+                return 1
+            return 2
+        traces.sort(key=priority)
+
+    os.makedirs(args.output_dir, exist_ok=True)
+
+    # Check for existing imported tasks to avoid duplicates
+    existing_run_ids = set()
+    for fname in os.listdir(args.output_dir):
+        if fname.endswith(".json"):
+            task = load_json(os.path.join(args.output_dir, fname))
+            if task and task.get("metadata", {}).get("langsmith_run_id"):
+                existing_run_ids.add(task["metadata"]["langsmith_run_id"])
+
+    imported = 0
+    skipped_no_input = 0
+    skipped_duplicate = 0
+    negative_count = 0
+
+    for run in traces:
+        if imported >= args.max_tasks:
+            break
+
+        run_id = str(run.get("id", ""))
+        if run_id in existing_run_ids:
+            skipped_duplicate += 1
+            continue
+
+        user_input = extract_input_from_trace(run)
+        if not user_input or len(user_input.strip()) < 5:
+            skipped_no_input += 1
+            continue
+
+        feedback = extract_feedback(run)
+        has_error = bool(run.get("error"))
+        task_id = f"{args.prefix}_{short_id(run_id)}"
+
+        task = {
+            "id": task_id,
+            "input": user_input.strip(),
+            "metadata": {
+                "difficulty": infer_difficulty(user_input),
+                "category": run.get("name", "unknown"),
+                "type": "production",
+                "source": "imported",
+                "langsmith_run_id": run_id,
+                "had_error": has_error,
+                "user_feedback": feedback,
+            },
+        }
+
+        out_path = os.path.join(args.output_dir, f"{task_id}.json")
+        with open(out_path, "w") as f:
+            json.dump(task, f, indent=2)
+
+        imported += 1
+        if feedback == "negative":
+            negative_count += 1
+
+    summary = {
+        "imported": imported,
+        "negative_feedback": negative_count,
+        "skipped_no_input": skipped_no_input,
+        "skipped_duplicate": skipped_duplicate,
+        "total_traces": len(traces),
+    }
+    print(json.dumps(summary))
+    print(f"Imported {imported} production traces as tasks ({negative_count} with negative feedback)")
+    if skipped_duplicate:
+        print(f"  Skipped {skipped_duplicate} already-imported traces")
+    if skipped_no_input:
+        print(f"  Skipped {skipped_no_input} traces with no extractable input")
+
+
+if __name__ == "__main__":
+    main()

package/tools/test_growth.py

@@ -0,0 +1,230 @@
+#!/usr/bin/env python3
+"""Test Suite Growth for Harness Evolver.
+
+Generates regression test tasks when previously-failing tasks are now passing.
+Creates mechanical variations of fixed tasks to prevent future regressions.
+
+Usage:
+    python3 test_growth.py \
+        --current-scores .harness-evolver/harnesses/v003/scores.json \
+        --previous-scores .harness-evolver/harnesses/v002/scores.json \
+        --tasks-dir .harness-evolver/eval/tasks/ \
+        --output-dir .harness-evolver/eval/tasks/ \
+        --max-total-tasks 60
+
+Stdlib-only. No external dependencies.
+"""
+
+import argparse
+import json
+import os
+import re
+import sys
+
+
+def load_json(path):
+    """Load JSON file, return None if missing or invalid."""
+    if not path or not os.path.exists(path):
+        return None
+    try:
+        with open(path) as f:
+            return json.load(f)
+    except (json.JSONDecodeError, OSError):
+        return None
+
+
+def find_fixed_tasks(current_scores, previous_scores, fix_threshold_before=0.5, fix_threshold_after=0.8):
+    """Find tasks that improved significantly: score < before_threshold → > after_threshold."""
+    current_per_task = current_scores.get("per_task", {})
+    previous_per_task = previous_scores.get("per_task", {})
+
+    fixed = []
+    for tid, curr_data in current_per_task.items():
+        if not isinstance(curr_data, dict):
+            continue
+        curr_score = curr_data.get("score", 0)
+        prev_data = previous_per_task.get(tid, {})
+        prev_score = prev_data.get("score", 0) if isinstance(prev_data, dict) else 0
+
+        if prev_score < fix_threshold_before and curr_score > fix_threshold_after:
+            fixed.append({
+                "task_id": tid,
+                "previous_score": prev_score,
+                "current_score": curr_score,
+                "improvement": curr_score - prev_score,
+            })
+
+    # Sort by improvement (biggest fixes first)
+    fixed.sort(key=lambda x: -x["improvement"])
+    return fixed
+
+
+def count_existing_tasks(directory):
+    """Count existing task JSON files in directory."""
+    if not os.path.isdir(directory):
+        return 0
+    return sum(1 for f in os.listdir(directory) if f.endswith(".json"))
+
+
+def next_regression_id(output_dir):
+    """Find the next available regression task ID."""
+    existing = set()
+    if os.path.isdir(output_dir):
+        for fname in os.listdir(output_dir):
+            m = re.match(r"regression_(\d+)\.json", fname)
+            if m:
+                existing.add(int(m.group(1)))
+    n = 1
+    while n in existing:
+        n += 1
+    return n
+
+
+def generate_variations(original_input, task_id):
+    """Generate 2-3 mechanical variations of an input string.
+
+    Uses simple string transforms — no LLM needed:
+    - Rephrase by reordering
+    - Add qualifying clause
+    - Simplify to minimal form
+    """
+    variations = []
+    text = original_input.strip()
+
+    # Variation 1: Add a qualifying clause
+    qualifiers = [
+        "Please be specific and detailed in your response.",
+        "Consider edge cases in your answer.",
+        "Provide a concise but thorough response.",
+        "Think step by step before answering.",
+    ]
+    # Pick qualifier based on hash of task_id for determinism
+    qi = hash(task_id) % len(qualifiers)
+    v1 = f"{text}\n\n{qualifiers[qi]}"
+    variations.append(("qualified", v1))
+
+    # Variation 2: Reorder sentences if multiple exist
+    sentences = re.split(r"(?<=[.!?])\s+", text)
+    if len(sentences) >= 2:
+        # Swap first two sentences
+        reordered = sentences[1:] + sentences[:1]
+        v2 = " ".join(reordered)
+        variations.append(("reordered", v2))
+    else:
+        # If single sentence, prepend "Given the context: "
+        v2 = f"Given the following context, {text[0].lower()}{text[1:]}" if len(text) > 1 else text
+        variations.append(("rephrased", v2))
+
+    # Variation 3: Minimal version — strip to core question
+    # Remove qualifiers, keep just the main ask
+    minimal = text
+    # Strip common padding phrases
+    for prefix in ["Please ", "Can you ", "Could you ", "I would like you to ", "I need you to "]:
+        if minimal.startswith(prefix):
+            minimal = minimal[len(prefix):]
+            minimal = minimal[0].upper() + minimal[1:] if minimal else minimal
+            break
+    if minimal != text:
+        variations.append(("minimal", minimal))
+
+    return variations
+
+
+def main():
+    parser = argparse.ArgumentParser(description="Generate regression test tasks from score improvements")
+    parser.add_argument("--current-scores", required=True, help="Path to current version's scores.json")
+    parser.add_argument("--previous-scores", required=True, help="Path to previous version's scores.json")
+    parser.add_argument("--tasks-dir", required=True, help="Path to eval/tasks/ (to read originals)")
+    parser.add_argument("--output-dir", required=True, help="Directory to write regression tasks")
+    parser.add_argument("--max-total-tasks", type=int, default=60, help="Cap total tasks in output-dir (default 60)")
+    args = parser.parse_args()
+
+    current = load_json(args.current_scores)
+    previous = load_json(args.previous_scores)
+
+    if not current or not previous:
+        print("Missing scores files — skipping test growth")
+        return
+
+    # Find tasks that were fixed
+    fixed = find_fixed_tasks(current, previous)
+    if not fixed:
+        print("No tasks improved significantly — no regression tasks needed")
+        return
+
+    # Check capacity
+    existing_count = count_existing_tasks(args.output_dir)
+    available_slots = args.max_total_tasks - existing_count
+    if available_slots <= 0:
+        print(f"Task suite already at capacity ({existing_count}/{args.max_total_tasks}) — skipping growth")
+        return
+
+    os.makedirs(args.output_dir, exist_ok=True)
+    regression_id = next_regression_id(args.output_dir)
+    tasks_added = 0
+    fixed_ids = []
+
+    for fix_info in fixed:
+        if tasks_added >= available_slots:
+            break
+
+        tid = fix_info["task_id"]
+        # Load original task
+        task_path = os.path.join(args.tasks_dir, f"{tid}.json")
+        original = load_json(task_path)
+        if not original:
+            continue
+
+        original_input = original.get("input", "")
+        if not original_input:
+            continue
+
+        original_meta = original.get("metadata", {})
+        variations = generate_variations(original_input, tid)
+
+        for var_type, var_input in variations:
+            if tasks_added >= available_slots:
+                break
+
+            reg_id = f"regression_{regression_id:03d}"
+            task = {
+                "id": reg_id,
+                "input": var_input,
+                "metadata": {
+                    "difficulty": original_meta.get("difficulty", "medium"),
+                    "category": original_meta.get("category", "unknown"),
+                    "type": "regression",
+                    "source": "regression",
+                    "regression_for": tid,
+                    "variation": var_type,
+                    "previous_score": fix_info["previous_score"],
+                    "fixed_at_score": fix_info["current_score"],
+                },
+            }
+
+            # Include expected if original had it
+            if "expected" in original:
+                task["expected"] = original["expected"]
+
+            out_path = os.path.join(args.output_dir, f"{reg_id}.json")
+            with open(out_path, "w") as f:
+                json.dump(task, f, indent=2)
+
+            tasks_added += 1
+            regression_id += 1
+
+        fixed_ids.append(tid)
+
+    # Output summary
+    summary = {
+        "tasks_added": tasks_added,
+        "fixed_tasks": fixed_ids,
+        "total_tasks_now": existing_count + tasks_added,
+        "max_total_tasks": args.max_total_tasks,
+    }
+    print(json.dumps(summary))
+    print(f"Added {tasks_added} regression tasks to lock in improvements on: {', '.join(fixed_ids)}")
+
+
+if __name__ == "__main__":
+    main()

package/tools/trace_insights.py

@@ -0,0 +1,350 @@
+#!/usr/bin/env python3
+"""Trace Insights Generator for Harness Evolver.
+
+Analyzes LangSmith traces + per-task scores to produce structured insights.
+Clusters errors, analyzes token usage, cross-references with scores,
+and generates data-driven hypotheses.
+
+Usage:
+    python3 trace_insights.py \
+        --langsmith-runs .harness-evolver/langsmith_runs.json \
+        --scores .harness-evolver/harnesses/v002/scores.json \
+        --tasks-dir .harness-evolver/eval/tasks/ \
+        --output .harness-evolver/trace_insights.json \
+        [--langsmith-stats .harness-evolver/langsmith_stats.json]
+
+Stdlib-only. No external dependencies.
+"""
+
+import argparse
+import json
+import os
+import sys
+from datetime import datetime, timezone
+
+
+def load_json(path):
+    """Load JSON file, return None if missing or invalid."""
+    if not path or not os.path.exists(path):
+        return None
+    try:
+        with open(path) as f:
+            return json.load(f)
+    except (json.JSONDecodeError, OSError):
+        return None
+
+
+def cluster_errors(runs):
+    """Group runs by error pattern (first 80 chars of error message)."""
+    clusters = {}
+    for run in runs:
+        error = run.get("error")
+        if not error:
+            continue
+        # Normalize: take first 80 chars, strip whitespace
+        pattern = error.strip()[:80]
+        clusters.setdefault(pattern, []).append(run)
+    return [
+        {"pattern": pattern, "count": len(runs_list), "run_names": [r.get("name", "?") for r in runs_list[:5]]}
+        for pattern, runs_list in sorted(clusters.items(), key=lambda x: -len(x[1]))
+    ]
+
+
+def analyze_tokens(runs):
+    """Bucket runs by token usage: low (<500), medium (500-2000), high (>2000)."""
+    buckets = {"low": [], "medium": [], "high": []}
+    for run in runs:
+        tokens = run.get("tokens") or run.get("total_tokens") or 0
+        if tokens < 500:
+            buckets["low"].append(run)
+        elif tokens < 2000:
+            buckets["medium"].append(run)
+        else:
+            buckets["high"].append(run)
+    return {
+        name: {"count": len(items), "avg_tokens": sum((r.get("tokens") or r.get("total_tokens") or 0) for r in items) / max(len(items), 1)}
+        for name, items in buckets.items()
+    }
+
+
+def analyze_responses(runs):
+    """Bucket runs by response length: empty, short (<100), normal (100-1000), long (>1000)."""
+    buckets = {"empty": [], "short": [], "normal": [], "long": []}
+    for run in runs:
+        resp = run.get("llm_response") or run.get("output") or ""
+        length = len(resp)
+        if length == 0:
+            buckets["empty"].append(run)
+        elif length < 100:
+            buckets["short"].append(run)
+        elif length < 1000:
+            buckets["normal"].append(run)
+        else:
+            buckets["long"].append(run)
+    return {
+        name: {"count": len(items)}
+        for name, items in buckets.items()
+        if items
+    }
+
+
+def cross_reference_scores(runs, scores_data, tasks_dir):
+    """Cross-reference trace patterns with per-task scores."""
+    per_task = scores_data.get("per_task", {}) if scores_data else {}
+    if not per_task:
+        return {}
+
+    # Load task metadata for category mapping
+    task_meta = {}
+    if tasks_dir and os.path.isdir(tasks_dir):
+        for fname in os.listdir(tasks_dir):
+            if fname.endswith(".json"):
+                path = os.path.join(tasks_dir, fname)
+                try:
+                    with open(path) as f:
+                        t = json.load(f)
+                    tid = t.get("id", fname.replace(".json", ""))
+                    task_meta[tid] = t.get("metadata", {})
+                except (json.JSONDecodeError, OSError):
+                    pass
+
+    # Score statistics
+    scores = [v.get("score", 0) for v in per_task.values() if isinstance(v, dict)]
+    if not scores:
+        return {}
+
+    failing = {tid: v for tid, v in per_task.items() if isinstance(v, dict) and v.get("score", 0) < 0.5}
+    passing = {tid: v for tid, v in per_task.items() if isinstance(v, dict) and v.get("score", 0) >= 0.8}
+
+    # Group failures by category
+    failure_categories = {}
+    for tid in failing:
+        meta = task_meta.get(tid, {})
+        cat = meta.get("category", meta.get("type", "unknown"))
+        failure_categories.setdefault(cat, []).append(tid)
+
+    return {
+        "total_tasks": len(per_task),
+        "avg_score": sum(scores) / len(scores),
+        "failing_count": len(failing),
+        "passing_count": len(passing),
+        "failing_task_ids": list(failing.keys()),
+        "failure_categories": {cat: tids for cat, tids in sorted(failure_categories.items(), key=lambda x: -len(x[1]))},
+    }
+
+
+def correlate_tokens_scores(runs, scores_data):
+    """Check if token usage correlates with task scores."""
+    per_task = scores_data.get("per_task", {}) if scores_data else {}
+    if not per_task or not runs:
+        return None
+
+    # Simple correlation: avg score for high-token vs low-token runs
+    token_scores = {"low": [], "medium": [], "high": []}
+    for run in runs:
+        tokens = run.get("tokens") or run.get("total_tokens") or 0
+        # Try to match run to task by name
+        name = run.get("name", "")
+        for tid, tdata in per_task.items():
+            if isinstance(tdata, dict) and tid in name:
+                score = tdata.get("score", 0)
+                if tokens < 500:
+                    token_scores["low"].append(score)
+                elif tokens < 2000:
+                    token_scores["medium"].append(score)
+                else:
+                    token_scores["high"].append(score)
+                break
+
+    result = {}
+    for bucket, scores in token_scores.items():
+        if scores:
+            result[bucket] = {"count": len(scores), "avg_score": sum(scores) / len(scores)}
+    return result if result else None
+
+
+def generate_hypotheses(error_clusters, token_analysis, response_analysis, score_cross_ref, token_score_corr):
+    """Generate data-driven hypotheses about failure patterns."""
+    hypotheses = []
+
+    # Hypothesis: errors cause failures
+    if error_clusters:
+        total_errors = sum(c["count"] for c in error_clusters)
+        top_error = error_clusters[0]
+        hypotheses.append(
+            f"{total_errors} runs had errors. Most common: \"{top_error['pattern']}\" ({top_error['count']} occurrences)"
+        )
+
+    # Hypothesis: empty responses
+    if response_analysis and response_analysis.get("empty", {}).get("count", 0) > 0:
+        n = response_analysis["empty"]["count"]
+        hypotheses.append(
+            f"{n} runs returned empty responses — possible API timeout, rate limiting, or invalid prompt"
+        )
+
+    # Hypothesis: high token usage correlates with low scores
+    if token_score_corr:
+        high = token_score_corr.get("high", {})
+        low = token_score_corr.get("low", {})
+        if high.get("avg_score", 1) < low.get("avg_score", 0) - 0.15:
+            hypotheses.append(
+                f"High-token runs avg score {high['avg_score']:.2f} vs low-token {low['avg_score']:.2f} — model may be verbose but inaccurate"
+            )
+
+    # Hypothesis: specific category failures
+    if score_cross_ref and score_cross_ref.get("failure_categories"):
+        cats = score_cross_ref["failure_categories"]
+        top_cat = next(iter(cats))
+        count = len(cats[top_cat])
+        hypotheses.append(
+            f"Category \"{top_cat}\" has {count} failing tasks — may need targeted prompt or tool improvement"
+        )
+
+    # Hypothesis: many failing
+    if score_cross_ref:
+        fail_count = score_cross_ref.get("failing_count", 0)
+        total = score_cross_ref.get("total_tasks", 1)
+        if fail_count > total * 0.5:
+            hypotheses.append(
+                f"{fail_count}/{total} tasks failing (>{50}%) — fundamental approach issue, not edge cases"
+            )
+
+    return hypotheses
+
+
+def identify_top_issues(error_clusters, response_analysis, score_cross_ref):
+    """Identify the most impactful issues sorted by severity."""
+    issues = []
+
+    # Empty responses = high severity
+    if response_analysis and response_analysis.get("empty", {}).get("count", 0) > 0:
+        issues.append({
+            "type": "empty_response",
+            "severity": "high",
+            "count": response_analysis["empty"]["count"],
+            "description": "Runs returning empty responses",
+        })
+
+    # Errors = high severity
+    if error_clusters:
+        for cluster in error_clusters[:3]:
+            issues.append({
+                "type": "error",
+                "severity": "high" if cluster["count"] > 2 else "medium",
+                "count": cluster["count"],
+                "pattern": cluster["pattern"],
+                "description": f"Error: {cluster['pattern'][:60]}",
+            })
+
+    # Category-concentrated failures = medium severity
+    if score_cross_ref and score_cross_ref.get("failure_categories"):
+        for cat, tids in list(score_cross_ref["failure_categories"].items())[:3]:
+            issues.append({
+                "type": "category_failure",
+                "severity": "medium" if len(tids) >= 3 else "low",
+                "category": cat,
+                "tasks": tids,
+                "description": f"Category \"{cat}\" has {len(tids)} failing tasks",
+            })
+
+    # Sort by severity
+    severity_order = {"high": 0, "medium": 1, "low": 2}
+    issues.sort(key=lambda x: severity_order.get(x.get("severity", "low"), 3))
+    return issues
+
+
+def main():
+    parser = argparse.ArgumentParser(description="Generate trace insights from LangSmith data + scores")
+    parser.add_argument("--langsmith-runs", required=True, help="Path to langsmith_runs.json")
+    parser.add_argument("--langsmith-stats", help="Path to langsmith_stats.json (optional)")
+    parser.add_argument("--scores", required=True, help="Path to best version's scores.json")
+    parser.add_argument("--tasks-dir", required=True, help="Path to eval/tasks/ directory")
+    parser.add_argument("--output", required=True, help="Output path for trace_insights.json")
+    args = parser.parse_args()
+
+    runs = load_json(args.langsmith_runs)
+    stats = load_json(args.langsmith_stats)
+    scores_data = load_json(args.scores)
+
+    if not runs and not scores_data:
+        # Nothing to analyze — write minimal insights
+        insights = {
+            "generated_at": datetime.now(timezone.utc).isoformat(),
+            "summary": "No trace data or scores available for analysis",
+            "error_clusters": [],
+            "token_analysis": {},
+            "response_analysis": {},
+            "hypotheses": [],
+            "top_issues": [],
+        }
+        os.makedirs(os.path.dirname(args.output) or ".", exist_ok=True)
+        with open(args.output, "w") as f:
+            json.dump(insights, f, indent=2)
+        print("No data available — wrote empty insights")
+        return
+
+    runs = runs or []
+
+    # Phase 1: Cluster traces
+    error_clusters = cluster_errors(runs)
+    token_analysis = analyze_tokens(runs)
+    response_analysis = analyze_responses(runs)
+
+    # Phase 2: Cross-reference with scores
+    score_cross_ref = cross_reference_scores(runs, scores_data, args.tasks_dir)
+    token_score_corr = correlate_tokens_scores(runs, scores_data)
+
+    # Phase 3: Generate hypotheses
+    hypotheses = generate_hypotheses(error_clusters, token_analysis, response_analysis, score_cross_ref, token_score_corr)
+
+    # Phase 4: Identify top issues
+    top_issues = identify_top_issues(error_clusters, response_analysis, score_cross_ref)
+
+    # Build summary line
+    parts = []
+    if error_clusters:
+        parts.append(f"{len(error_clusters)} error pattern(s)")
+    if score_cross_ref:
+        parts.append(f"{score_cross_ref.get('failing_count', 0)}/{score_cross_ref.get('total_tasks', 0)} tasks failing")
+        parts.append(f"avg score {score_cross_ref.get('avg_score', 0):.2f}")
+    summary = "; ".join(parts) if parts else "Analysis complete, no major issues found"
+
+    # Merge stats if available
+    stats_summary = {}
+    if stats:
+        stats_summary = {
+            "total_runs": stats.get("total_runs") or stats.get("run_count"),
+            "error_rate": stats.get("error_rate"),
+            "avg_latency_ms": stats.get("avg_latency_ms") or stats.get("latency_p50"),
+            "p95_latency_ms": stats.get("latency_p95"),
+            "avg_tokens": stats.get("avg_tokens") or stats.get("avg_total_tokens"),
+        }
+        # Remove None values
+        stats_summary = {k: v for k, v in stats_summary.items() if v is not None}
+
+    insights = {
+        "generated_at": datetime.now(timezone.utc).isoformat(),
+        "summary": summary,
+        "langsmith_stats": stats_summary if stats_summary else None,
+        "error_clusters": error_clusters,
+        "token_analysis": token_analysis,
+        "response_analysis": response_analysis,
+        "score_cross_ref": score_cross_ref if score_cross_ref else None,
+        "token_score_correlation": token_score_corr,
+        "hypotheses": hypotheses,
+        "top_issues": top_issues,
+    }
+
+    # Remove None values at top level
+    insights = {k: v for k, v in insights.items() if v is not None}
+
+    os.makedirs(os.path.dirname(args.output) or ".", exist_ok=True)
+    with open(args.output, "w") as f:
+        json.dump(insights, f, indent=2)
+
+    print(f"Trace insights generated: {summary}")
+    print(f"  {len(error_clusters)} error cluster(s), {len(hypotheses)} hypothesis(es), {len(top_issues)} issue(s)")
+
+
+if __name__ == "__main__":
+    main()