harness-evolver 2.9.1 → 3.0.0

package/skills/architect/SKILL.md

@@ -1,93 +0,0 @@
----
-name: harness-evolver:architect
-description: "Use when the user wants to analyze harness architecture, get a topology recommendation, understand if their agent pattern is optimal, or after stagnation in the evolution loop."
-argument-hint: "[--force]"
-allowed-tools: [Read, Write, Edit, Bash, Glob, Grep, Agent]
----
-
-# /harness-evolver:architect
-
-Analyze the current harness architecture and recommend the optimal multi-agent topology.
-
-## Prerequisites
-
-`.harness-evolver/` must exist. If not, tell user to run `harness-evolver:init` first.
-
-```bash
-if [ ! -d ".harness-evolver" ]; then
-  echo "ERROR: .harness-evolver/ not found. Run /harness-evolver:init first."
-  exit 1
-fi
-```
-
-## Resolve Tool Path
-
-```bash
-TOOLS=$([ -d ".harness-evolver/tools" ] && echo ".harness-evolver/tools" || echo "$HOME/.harness-evolver/tools")
-```
-
-Use `$TOOLS` prefix for all tool calls below.
-
-## What To Do
-
-1. Check `.harness-evolver/` exists.
-
-2. Run architecture analysis tool:
-```bash
-python3 $TOOLS/analyze_architecture.py \
-    --harness .harness-evolver/baseline/harness.py \
-    -o .harness-evolver/architecture_signals.json
-```
-
-If evolution has run, add trace and score data:
-```bash
-python3 $TOOLS/analyze_architecture.py \
-    --harness .harness-evolver/harnesses/{best}/harness.py \
-    --traces-dir .harness-evolver/harnesses/{best}/traces \
-    --summary .harness-evolver/summary.json \
-    -o .harness-evolver/architecture_signals.json
-```
-
-3. Dispatch using the Agent tool with `subagent_type: "harness-evolver-architect"`:
-
-```
-Agent(
-  subagent_type: "harness-evolver-architect",
-  description: "Architect: topology analysis",
-  prompt: |
-    <objective>
-    Analyze the harness architecture and recommend the optimal multi-agent topology.
-    {If called from evolve: "The evolution loop stagnated/regressed after N iterations."}
-    {If called by user: "The user requested an architecture analysis."}
-    </objective>
-
-    <files_to_read>
-    - .harness-evolver/architecture_signals.json
-    - .harness-evolver/config.json
-    - .harness-evolver/baseline/harness.py
-    - .harness-evolver/summary.json (if exists)
-    - .harness-evolver/PROPOSER_HISTORY.md (if exists)
-    </files_to_read>
-
-    <output>
-    Write:
-    - .harness-evolver/architecture.json
-    - .harness-evolver/architecture.md
-    </output>
-
-    <success_criteria>
-    - Classifies current topology correctly
-    - Recommendation includes migration path with concrete steps
-    - Considers detected stack and API key availability
-    - Confidence rating is honest (low/medium/high)
-    </success_criteria>
-)
-```
-
-4. Wait for `## ARCHITECTURE ANALYSIS COMPLETE`.
-
-5. Print summary: current -> recommended, confidence, migration steps.
-
-## Arguments
-
-- `--force` — re-run analysis even if `architecture.json` already exists. Without this flag, if `architecture.json` exists, just display the existing recommendation.

package/skills/compare/SKILL.md

@@ -1,73 +0,0 @@
----
-name: harness-evolver:compare
-description: "Use when the user wants to compare two harness versions, understand what changed between iterations, see why one version scored better than another, or debug a regression."
-argument-hint: "<vA> <vB>"
-allowed-tools: [Read, Bash, Glob, Grep]
----
-
-# /harness-evolver:compare
-
-Compare two harness versions side by side.
-
-## Arguments
-
-- `vA` — first version (e.g., `v001`, `baseline`)
-- `vB` — second version (e.g., `v003`)
-
-If only one version given, compare it against the current best.
-If no versions given, compare the two most recent.
-
-## What To Do
-
-### 1. Code Diff
-
-```bash
-diff .harness-evolver/harnesses/{vA}/harness.py .harness-evolver/harnesses/{vB}/harness.py
-```
-
-If config changed:
-```bash
-diff .harness-evolver/harnesses/{vA}/config.json .harness-evolver/harnesses/{vB}/config.json
-```
-
-### 2. Score Comparison
-
-```bash
-cat .harness-evolver/harnesses/{vA}/scores.json
-cat .harness-evolver/harnesses/{vB}/scores.json
-```
-
-Report: combined_score delta, per-task wins/losses.
-
-### 3. Per-Task Analysis
-
-For tasks where scores diverge, show what each version produced:
-
-```bash
-cat .harness-evolver/harnesses/{vA}/traces/task_{ID}/output.json
-cat .harness-evolver/harnesses/{vB}/traces/task_{ID}/output.json
-```
-
-### 4. Proposal Context
-
-```bash
-cat .harness-evolver/harnesses/{vB}/proposal.md
-```
-
-Show what the proposer intended and whether the result matched expectations.
-
-## Report Format
-
-```
-v001 (0.62) vs v003 (0.71) — +0.09 improvement
-
-Code changes:
-  + Added few-shot examples (3 examples)
-  ~ Changed prompt template
-  - Removed retry logic
-
-Per-task:
-  task_001: 1.0 → 1.0 (unchanged)
-  task_007: 0.0 → 1.0 (FIXED — was cardiac, now correctly classified)
-  task_008: 1.0 → 0.0 (REGRESSION — was neurological, now wrong)
-```

package/skills/critic/SKILL.md

@@ -1,67 +0,0 @@
----
-name: harness-evolver:critic
-description: "Use when scores converge suspiciously fast, eval quality is questionable, the harness reaches 1.0 in few iterations, or the user wants to validate that improvements are genuine. Also triggers automatically when score jumps >0.3 in one iteration."
-allowed-tools: [Read, Write, Edit, Bash, Glob, Grep, Agent]
----
-
-# /harness-evolver:critic
-
-Analyze eval quality and detect eval gaming.
-
-## Resolve Tool Path
-
-```bash
-TOOLS=$([ -d ".harness-evolver/tools" ] && echo ".harness-evolver/tools" || echo "$HOME/.harness-evolver/tools")
-```
-
-## Prerequisites
-
-`.harness-evolver/` must exist with at least one evaluated version (v001+).
-
-## What To Do
-
-1. Read `summary.json` and identify the suspicious pattern (score jump, premature convergence).
-
-2. Dispatch using the Agent tool with `subagent_type: "harness-evolver-critic"`:
-
-```
-Agent(
-  subagent_type: "harness-evolver-critic",
-  description: "Critic: analyze eval quality",
-  prompt: |
-    <objective>
-    Analyze eval quality for this harness evolution project.
-    The best version is {version} with score {score} achieved in {iterations} iteration(s).
-    {Specific concern: "Score jumped from X to Y in one iteration" or "Perfect score in N iterations"}
-    </objective>
-
-    <files_to_read>
-    - .harness-evolver/eval/eval.py
-    - .harness-evolver/summary.json
-    - .harness-evolver/harnesses/{best_version}/scores.json
-    - .harness-evolver/harnesses/{best_version}/harness.py
-    - .harness-evolver/harnesses/{best_version}/proposal.md
-    - .harness-evolver/config.json
-    - .harness-evolver/langsmith_stats.json (if exists)
-    </files_to_read>
-
-    <output>
-    Write:
-    - .harness-evolver/critic_report.md (human-readable analysis)
-    - .harness-evolver/eval/eval_improved.py (if weaknesses found)
-    </output>
-
-    <success_criteria>
-    - Identifies specific weaknesses in eval.py with examples
-    - If gaming detected, shows exact tasks/outputs that expose the weakness
-    - Improved eval preserves the --results-dir/--tasks-dir/--scores interface
-    - Re-scores the best version with improved eval to quantify the difference
-    </success_criteria>
-)
-```
-
-3. Wait for `## CRITIC REPORT COMPLETE`.
-
-4. Report findings to user. If `eval_improved.py` was written:
-   - Show score comparison (current eval vs improved eval)
-   - Ask: "Adopt the improved eval? This will affect future iterations."

package/skills/diagnose/SKILL.md

@@ -1,96 +0,0 @@
----
-name: harness-evolver:diagnose
-description: "Use when the user wants to understand why a specific harness version failed, investigate a regression, analyze trace data, or debug a low score. Also use when the user says 'why did v003 fail' or 'what went wrong'."
-argument-hint: "[version]"
-allowed-tools: [Read, Bash, Glob, Grep]
----
-
-# /harness-evolver:diagnose
-
-Deep analysis of a harness version's execution traces and scores.
-
-## Arguments
-
-- `version` — version to diagnose (e.g., `v003`). If not given, diagnose the worst or most recent regression.
-
-## Resolve Tool Path
-
-```bash
-TOOLS=$([ -d ".harness-evolver/tools" ] && echo ".harness-evolver/tools" || echo "$HOME/.harness-evolver/tools")
-```
-
-## What To Do
-
-### 1. Identify the Version
-
-If not specified, find the worst or most recent regression:
-
-```bash
-python3 $TOOLS/state.py show --base-dir .harness-evolver
-cat .harness-evolver/summary.json
-```
-
-### 2. Score Breakdown
-
-```bash
-cat .harness-evolver/harnesses/{version}/scores.json
-```
-
-Identify which tasks failed (`score: 0.0`) and which passed.
-
-### 3. Trace Analysis (failed tasks)
-
-For each failed task:
-
-```bash
-cat .harness-evolver/harnesses/{version}/traces/{task_id}/input.json
-cat .harness-evolver/harnesses/{version}/traces/{task_id}/output.json
-```
-
-Look for patterns: wrong format? wrong category? empty output? crash?
-
-### 4. Error Search
-
-```bash
-grep -r "error\|Error\|FAIL\|exception\|Traceback" .harness-evolver/harnesses/{version}/traces/
-cat .harness-evolver/harnesses/{version}/traces/stderr.log
-```
-
-### 5. Compare with Parent
-
-Read the proposal to find the parent version:
-
-```bash
-cat .harness-evolver/harnesses/{version}/proposal.md
-```
-
-Then diff:
-
-```bash
-diff .harness-evolver/harnesses/{parent}/harness.py .harness-evolver/harnesses/{version}/harness.py
-```
-
-### 6. LangSmith (if available)
-
-If `langsmith-cli` is installed and LangSmith is configured:
-
-```bash
-langsmith-cli --json runs list --project harness-evolver-{version} --failed --fields id,name,error,inputs
-langsmith-cli --json runs stats --project harness-evolver-{version}
-```
-
-### 7. Report
-
-```
-Diagnosis: v003 (score: 0.31) — REGRESSION from v001 (0.62)
-
-Root cause: Prompt template change broke JSON parsing
-  - 4/10 tasks returned malformed output
-  - stderr shows: json.JSONDecodeError on 4 tasks
-  - The change on line 42 removed the "Reply with ONLY..." instruction
-
-Affected tasks: task_002, task_005, task_007, task_010
-Unaffected tasks: task_001, task_003, task_004, task_006, task_008, task_009
-
-Recommendation: Revert the prompt change, keep the retry logic from v002
-```

package/skills/import-traces/SKILL.md

@@ -1,102 +0,0 @@
----
-name: harness-evolver:import-traces
-description: "Use when the user wants to import real production traces from LangSmith as test tasks, convert traces to eval tasks, enrich their eval set with real-world data, or pull production data into their harness evaluation."
-argument-hint: "[--project NAME] [--limit N]"
-allowed-tools: [Read, Write, Edit, Bash, Glob, Grep]
----
-
-# /harness-evolver:import-traces
-
-Import production traces from LangSmith and convert them into eval tasks. This enriches the test suite with real-world inputs, prioritizing traces with negative user feedback.
-
-## Prerequisites
-
-- `.harness-evolver/` must exist. If not, tell user to run `harness-evolver:init` first.
-- `langsmith-cli` must be available. Check:
-
-```bash
-which langsmith-cli 2>/dev/null
-```
-
-If not found: "Install langsmith-cli first: `uv tool install langsmith-cli && langsmith-cli auth login`"
-
-## Resolve Tool Path
-
-```bash
-TOOLS=$([ -d ".harness-evolver/tools" ] && echo ".harness-evolver/tools" || echo "$HOME/.harness-evolver/tools")
-```
-
-## Parse Arguments
-
-- `--project NAME` — LangSmith project name (if not provided, discover interactively)
-- `--limit N` — max traces to import (default: 20)
-
-## Phase 1: Discover Projects
-
-If `--project` not provided, list available projects:
-
-```bash
-langsmith-cli --json projects list --limit 20 2>/dev/null
-```
-
-Show the user a list of projects with run counts. Let them pick one, or use the most recent.
-
-If `--project` is provided, use it directly.
-
-## Phase 2: Fetch Traces
-
-```bash
-langsmith-cli --json runs list \
-    --project "{project_name}" \
-    --limit {limit} \
-    --fields id,name,inputs,outputs,error,feedback_stats,total_tokens \
-    > /tmp/harness_import_traces.json 2>/dev/null
-```
-
-Check the output has data:
-```bash
-python3 -c "import json; data=json.load(open('/tmp/harness_import_traces.json')); print(f'{len(data)} traces fetched')"
-```
-
-If no traces found, tell user the project may be empty or the name may be wrong.
-
-## Phase 3: Convert to Tasks
-
-```bash
-python3 $TOOLS/import_traces.py \
-    --traces-json /tmp/harness_import_traces.json \
-    --output-dir .harness-evolver/eval/tasks/ \
-    --prefix imported \
-    --max-tasks {limit}
-```
-
-## Phase 4: Report
-
-Read the tool output and report:
-- How many traces were imported
-- How many had negative feedback (high priority)
-- How many were skipped (no extractable input, duplicates)
-- Total tasks now in eval set
-
-```bash
-ls .harness-evolver/eval/tasks/*.json | wc -l
-```
-
-Print:
-```
-Imported {N} production traces as eval tasks.
-  {M} with negative user feedback (high priority)
-  {K} skipped (no input or duplicates)
-  Total eval tasks: {total}
-
-Next: run `harness-evolver:evolve` to optimize against real-world inputs.
-```
-
-## Gotchas
-
-- Traces with no extractable user input are skipped (e.g., system-only runs)
-- Duplicate traces (same run ID) are automatically skipped
-- Imported tasks are tagged with `metadata.source: "imported"` and `metadata.type: "production"`
-- Tasks with negative feedback get `metadata.user_feedback: "negative"` — the proposer should prioritize these
-- The `metadata.langsmith_run_id` field links back to the original trace for debugging
-- Cleanup: `rm /tmp/harness_import_traces.json` after import