harness-evolver 2.8.1 → 2.9.0

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "harness-evolver",
-  "version": "2.8.1",
+  "version": "2.9.0",
   "description": "Meta-Harness-style autonomous harness optimization for Claude Code",
   "author": "Raphael Valdetaro",
   "license": "MIT",

package/skills/deploy/SKILL.md

@@ -2,7 +2,7 @@
 name: harness-evolver:deploy
 description: "Use when the user wants to use the best evolved harness in their project, promote a version to production, copy the winning harness back, or is done evolving and wants to apply the result."
 argument-hint: "[version]"
-allowed-tools: [Read, Write, Bash, Glob]
+allowed-tools: [Read, Write, Bash, Glob, AskUserQuestion]
 ---
 
 # /harness-evolver:deploy
@@ -32,19 +32,48 @@ cat .harness-evolver/harnesses/{version}/scores.json
 
 Report: version, score, improvement over baseline, what changed.
 
-### 3. Ask for Confirmation
+### 3. Ask Deploy Options (Interactive)
 
-> Deploy `{version}` (score: {score}, +{delta} over baseline) to your project?
-> This will copy `harness.py` and `config.json` to the project root.
+Use AskUserQuestion with TWO questions:
+
+```
+Question 1: "Where should the evolved harness go?"
+Header: "Deploy to"
+Options:
+  - "Overwrite original" — Replace {original_harness_path} with the evolved version
+  - "Copy to new file" — Save as harness_evolved.py alongside the original
+  - "Just show the diff" — Don't copy anything, just show what changed
+```
+
+```
+Question 2 (ONLY if user chose "Overwrite original"):
+"Back up the current harness before overwriting?"
+Header: "Backup"
+Options:
+  - "Yes, backup first" — Save current as {harness}.bak before overwriting
+  - "No, just overwrite" — Replace directly (git history has the original)
+```
 
 ### 4. Copy Files
 
+Based on the user's choices:
+
+**If "Overwrite original"**:
+- If backup: `cp {original_harness} {original_harness}.bak`
+- Then: `cp .harness-evolver/harnesses/{version}/harness.py {original_harness}`
+- Copy config.json if exists
+
+**If "Copy to new file"**:
 ```bash
-cp .harness-evolver/harnesses/{version}/harness.py ./harness.py
-cp .harness-evolver/harnesses/{version}/config.json ./config.json  # if exists
+cp .harness-evolver/harnesses/{version}/harness.py ./harness_evolved.py
+cp .harness-evolver/harnesses/{version}/config.json ./config_evolved.json  # if exists
 ```
 
-If the original entry point had a different name (e.g., `graph.py`), ask the user where to put it.
+**If "Just show the diff"**:
+```bash
+diff {original_harness} .harness-evolver/harnesses/{version}/harness.py
+```
+Do not copy anything.
 
 ### 5. Report
 

package/skills/evolve/SKILL.md

@@ -2,7 +2,7 @@
 name: harness-evolver:evolve
 description: "Use when the user wants to run the optimization loop, improve harness performance, evolve the harness, or iterate on harness quality. Requires .harness-evolver/ to exist (run harness-evolver:init first)."
 argument-hint: "[--iterations N]"
-allowed-tools: [Read, Write, Edit, Bash, Glob, Grep, Agent]
+allowed-tools: [Read, Write, Edit, Bash, Glob, Grep, Agent, AskUserQuestion]
 ---
 
 # /harness-evolver:evolve
@@ -24,6 +24,35 @@ TOOLS=$([ -d ".harness-evolver/tools" ] && echo ".harness-evolver/tools" || echo
 - `--iterations N` (default: 10)
 - Read `config.json` for `evolution.stagnation_limit` (default: 3) and `evolution.target_score`
 
+## Pre-Loop: Interactive Configuration
+
+If no `--iterations` argument was provided, ask the user interactively:
+
+Use AskUserQuestion with TWO questions:
+
+```
+Question 1: "How many evolution iterations?"
+Header: "Iterations"
+Options:
+  - "3 (quick)" — Fast exploration, good for testing setup
+  - "5 (balanced)" — Good trade-off between speed and quality
+  - "10 (thorough)" — Deep optimization, takes longer
+
+Question 2: "Stop early if score reaches?"
+Header: "Target"
+Options:
+  - "0.8 (good enough)" — Stop when the harness is reasonably good
+  - "0.9 (high quality)" — Stop when quality is high
+  - "0.95 (near perfect)" — Push for near-perfect scores
+  - "No limit" — Run all iterations regardless of score
+```
+
+Apply the answers:
+- Set iterations from question 1 (3, 5, or 10)
+- Set target_score from question 2 (0.8, 0.9, 0.95, or None)
+
+If `--iterations` WAS provided as argument, skip these questions and use the argument value.
+
 ## The Loop
 
 For each iteration:

package/skills/init/SKILL.md

@@ -2,7 +2,7 @@
 name: harness-evolver:init
 description: "Use when the user wants to set up harness optimization in their project, optimize an LLM agent, improve a harness, or mentions harness-evolver for the first time in a project without .harness-evolver/ directory."
 argument-hint: "[directory]"
-allowed-tools: [Read, Write, Edit, Bash, Glob, Grep, Agent]
+allowed-tools: [Read, Write, Edit, Bash, Glob, Grep, Agent, AskUserQuestion]
 ---
 
 # /harness-evolve-init
@@ -30,15 +30,77 @@ Look for:
 - Existing tasks: directories with JSON files containing `id` + `input` fields
 - Config: `config.json`, `config.yaml`, `.env`
 
+## Phase 1.5: Confirm Detection (Interactive)
+
+After scanning, present what was found and ask the user to confirm before proceeding.
+
+Use AskUserQuestion:
+
+```
+Question: "Here's what I detected. Does this look right?"
+Header: "Confirm"
+Options:
+  - "Looks good, proceed" — Continue with detected paths
+  - "Let me adjust paths" — User will provide correct paths
+  - "Start over in different directory" — Abort and let user cd elsewhere
+
+Show in the question description:
+  - Harness: {path or "not found"}
+  - Eval: {path or "not found — will use LLM-as-judge"}
+  - Tasks: {path with N files, or "not found — will generate"}
+  - Stack: {detected frameworks or "none detected"}
+  - Architecture: {topology or "unknown"}
+```
+
+If user chose "Let me adjust paths", ask which paths to change and update accordingly.
+
+## Phase 1.8: Eval Mode (Interactive — only if NO eval found)
+
+If no eval.py was detected, ask the user which evaluation mode to use:
+
+```
+Question: "No eval script found. How should outputs be scored?"
+Header: "Eval mode"
+Options:
+  - "LLM-as-judge (zero-config)" — Claude Code scores outputs on accuracy, completeness, relevance, hallucination. No expected answers needed.
+  - "Keyword matching" — Simple string matching against expected answers. Requires 'expected' field in tasks.
+  - "I'll provide my own eval.py" — Pause and let user create their eval script.
+```
+
+If "LLM-as-judge": copy eval_passthrough.py as eval.py.
+If "Keyword matching": create a simple keyword eval (check if expected substrings appear in output).
+If "I'll provide my own": print instructions for the eval contract and wait.
+
+## Phase 1.9: LangSmith Project (Interactive — only if LANGSMITH_API_KEY detected)
+
+If a LangSmith API key is available, discover projects and ask which one has production traces:
+
+```bash
+langsmith-cli --json projects list --limit 10 2>/dev/null
+```
+
+Use AskUserQuestion:
+```
+Question: "LangSmith detected. Which project has your production traces?"
+Header: "LangSmith"
+Options: (build from discovered projects — pick top 3-4 by recent activity)
+  - "{project_name_1}" — {run_count} runs, last active {date}
+  - "{project_name_2}" — {run_count} runs, last active {date}
+  - "{project_name_3}" — {run_count} runs, last active {date}
+  - "Skip — don't use production traces" — Proceed without production data
+```
+
+If a project is selected, pass it as `--langsmith-project` to init.py.
+
 ## Phase 2: Create What's Missing
 
 Three artifacts needed. For each — use existing if found, create if not.
 
 **Harness** (`harness.py`): If user's entry point doesn't match our CLI interface (`--input`, `--output`, `--traces-dir`, `--config`), create a thin wrapper that imports their code. Read their entry point first to understand the I/O format. Ask if unsure.
 
-**Eval** (`eval.py`): If an eval script exists, use it.
+**Eval** (`eval.py`): If an eval script exists, use it. If the user already chose an eval mode in Phase 1.8, follow that choice.
 
-If NO eval exists:
+If NO eval exists and no mode was chosen yet:
 - Copy `eval_passthrough.py` from `$TOOLS/eval_passthrough.py` as the project's eval.py:
   ```bash
   cp $TOOLS/eval_passthrough.py eval.py