harness-evolver 2.8.1 → 2.9.1

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "harness-evolver",
-  "version": "2.8.1",
+  "version": "2.9.1",
   "description": "Meta-Harness-style autonomous harness optimization for Claude Code",
   "author": "Raphael Valdetaro",
   "license": "MIT",

package/skills/deploy/SKILL.md

@@ -2,7 +2,7 @@
 name: harness-evolver:deploy
 description: "Use when the user wants to use the best evolved harness in their project, promote a version to production, copy the winning harness back, or is done evolving and wants to apply the result."
 argument-hint: "[version]"
-allowed-tools: [Read, Write, Bash, Glob]
+allowed-tools: [Read, Write, Bash, Glob, AskUserQuestion]
 ---
 
 # /harness-evolver:deploy
@@ -32,19 +32,48 @@ cat .harness-evolver/harnesses/{version}/scores.json
 
 Report: version, score, improvement over baseline, what changed.
 
-### 3. Ask for Confirmation
+### 3. Ask Deploy Options (Interactive)
 
-> Deploy `{version}` (score: {score}, +{delta} over baseline) to your project?
-> This will copy `harness.py` and `config.json` to the project root.
+Use AskUserQuestion with TWO questions:
+
+```
+Question 1: "Where should the evolved harness go?"
+Header: "Deploy to"
+Options:
+  - "Overwrite original" — Replace {original_harness_path} with the evolved version
+  - "Copy to new file" — Save as harness_evolved.py alongside the original
+  - "Just show the diff" — Don't copy anything, just show what changed
+```
+
+```
+Question 2 (ONLY if user chose "Overwrite original"):
+"Back up the current harness before overwriting?"
+Header: "Backup"
+Options:
+  - "Yes, backup first" — Save current as {harness}.bak before overwriting
+  - "No, just overwrite" — Replace directly (git history has the original)
+```
 
 ### 4. Copy Files
 
+Based on the user's choices:
+
+**If "Overwrite original"**:
+- If backup: `cp {original_harness} {original_harness}.bak`
+- Then: `cp .harness-evolver/harnesses/{version}/harness.py {original_harness}`
+- Copy config.json if exists
+
+**If "Copy to new file"**:
 ```bash
-cp .harness-evolver/harnesses/{version}/harness.py ./harness.py
-cp .harness-evolver/harnesses/{version}/config.json ./config.json  # if exists
+cp .harness-evolver/harnesses/{version}/harness.py ./harness_evolved.py
+cp .harness-evolver/harnesses/{version}/config.json ./config_evolved.json  # if exists
 ```
 
-If the original entry point had a different name (e.g., `graph.py`), ask the user where to put it.
+**If "Just show the diff"**:
+```bash
+diff {original_harness} .harness-evolver/harnesses/{version}/harness.py
+```
+Do not copy anything.
 
 ### 5. Report
 

package/skills/evolve/SKILL.md

@@ -2,7 +2,7 @@
 name: harness-evolver:evolve
 description: "Use when the user wants to run the optimization loop, improve harness performance, evolve the harness, or iterate on harness quality. Requires .harness-evolver/ to exist (run harness-evolver:init first)."
 argument-hint: "[--iterations N]"
-allowed-tools: [Read, Write, Edit, Bash, Glob, Grep, Agent]
+allowed-tools: [Read, Write, Edit, Bash, Glob, Grep, Agent, AskUserQuestion]
 ---
 
 # /harness-evolver:evolve
@@ -24,6 +24,46 @@ TOOLS=$([ -d ".harness-evolver/tools" ] && echo ".harness-evolver/tools" || echo
 - `--iterations N` (default: 10)
 - Read `config.json` for `evolution.stagnation_limit` (default: 3) and `evolution.target_score`
 
+## Pre-Loop: Interactive Configuration
+
+If no `--iterations` argument was provided, ask the user interactively:
+
+Use AskUserQuestion with TWO questions in a single call (simple single-select, no preview needed):
+
+```json
+{
+  "questions": [
+    {
+      "question": "How many evolution iterations?",
+      "header": "Iterations",
+      "multiSelect": false,
+      "options": [
+        {"label": "3 (quick)", "description": "Fast exploration, good for testing setup. ~15 min."},
+        {"label": "5 (balanced)", "description": "Good trade-off between speed and quality. ~30 min."},
+        {"label": "10 (thorough)", "description": "Deep optimization with adaptive strategies. ~1 hour."}
+      ]
+    },
+    {
+      "question": "Stop early if score reaches?",
+      "header": "Target",
+      "multiSelect": false,
+      "options": [
+        {"label": "0.8 (good enough)", "description": "Stop when the harness is reasonably good"},
+        {"label": "0.9 (high quality)", "description": "Stop when quality is high"},
+        {"label": "0.95 (near perfect)", "description": "Push for near-perfect scores"},
+        {"label": "No limit", "description": "Run all iterations regardless of score"}
+      ]
+    }
+  ]
+}
+```
+
+Apply the answers:
+- Set iterations from question 1 (3, 5, or 10)
+- Set target_score from question 2 (0.8, 0.9, 0.95, or None)
+
+If `--iterations` WAS provided as argument, skip these questions and use the argument value.
+
 ## The Loop
 
 For each iteration:

package/skills/init/SKILL.md

@@ -2,7 +2,7 @@
 name: harness-evolver:init
 description: "Use when the user wants to set up harness optimization in their project, optimize an LLM agent, improve a harness, or mentions harness-evolver for the first time in a project without .harness-evolver/ directory."
 argument-hint: "[directory]"
-allowed-tools: [Read, Write, Edit, Bash, Glob, Grep, Agent]
+allowed-tools: [Read, Write, Edit, Bash, Glob, Grep, Agent, AskUserQuestion]
 ---
 
 # /harness-evolve-init
@@ -30,15 +30,117 @@ Look for:
 - Existing tasks: directories with JSON files containing `id` + `input` fields
 - Config: `config.json`, `config.yaml`, `.env`
 
+## Phase 1.5: Confirm Detection (Interactive)
+
+After scanning, present what was found and ask the user to confirm before proceeding.
+
+Use AskUserQuestion:
+
+```
+Question: "Here's what I detected. Does this look right?"
+Header: "Confirm"
+Options:
+  - "Looks good, proceed" — Continue with detected paths
+  - "Let me adjust paths" — User will provide correct paths
+  - "Start over in different directory" — Abort and let user cd elsewhere
+
+Show in the question description:
+  - Harness: {path or "not found"}
+  - Eval: {path or "not found — will use LLM-as-judge"}
+  - Tasks: {path with N files, or "not found — will generate"}
+  - Stack: {detected frameworks or "none detected"}
+  - Architecture: {topology or "unknown"}
+```
+
+If user chose "Let me adjust paths", ask which paths to change and update accordingly.
+
+## Phase 1.8: Eval Mode (Interactive — only if NO eval found)
+
+If no eval.py was detected, ask the user which evaluation mode to use.
+
+Use AskUserQuestion with **preview** (single-select with side-by-side preview):
+
+```json
+{
+  "questions": [{
+    "question": "No eval script found. How should outputs be scored?",
+    "header": "Eval mode",
+    "multiSelect": false,
+    "options": [
+      {
+        "label": "LLM-as-judge (zero-config)",
+        "description": "Claude Code scores outputs automatically. No expected answers needed.",
+        "preview": "## LLM-as-Judge\n\nScoring dimensions:\n- **Accuracy** (40%) — correctness of output\n- **Completeness** (20%) — covers all aspects\n- **Relevance** (20%) — focused on the question\n- **No-Hallucination** (20%) — supported by facts\n\nEach scored 1-5, normalized to 0.0-1.0.\n\n**Requirements:** None. Works with any task format.\n\n```json\n{\"id\": \"task_001\", \"input\": \"your question\"}\n```"
+      },
+      {
+        "label": "Keyword matching",
+        "description": "Check if expected substrings appear in the output. Requires 'expected' field.",
+        "preview": "## Keyword Matching\n\nSimple deterministic scoring:\n- Score 1.0 if ALL expected keywords found in output\n- Score 0.0 otherwise\n\n**Requirements:** Tasks must include `expected` field:\n\n```json\n{\n  \"id\": \"task_001\",\n  \"input\": \"What is the capital of France?\",\n  \"expected\": \"Paris\"\n}\n```\n\nFast, deterministic, no LLM calls during eval."
+      },
+      {
+        "label": "I'll provide my own eval.py",
+        "description": "Pause setup. You write the eval script following the contract.",
+        "preview": "## Custom Eval Contract\n\nYour eval.py must accept:\n```\npython3 eval.py \\\n  --results-dir DIR \\\n  --tasks-dir DIR \\\n  --scores OUTPUT.json\n```\n\nMust write scores.json:\n```json\n{\n  \"combined_score\": 0.85,\n  \"per_task\": {\n    \"task_001\": {\"score\": 0.9},\n    \"task_002\": {\"score\": 0.8}\n  }\n}\n```\n\nScores must be 0.0 to 1.0."
+      }
+    ]
+  }]
+}
+```
+
+If "LLM-as-judge": copy eval_passthrough.py as eval.py.
+If "Keyword matching": create a simple keyword eval (check if expected substrings appear in output).
+If "I'll provide my own": print instructions for the eval contract and wait.
+
+## Phase 1.9: LangSmith Project (Interactive — only if LANGSMITH_API_KEY detected)
+
+If a LangSmith API key is available, discover projects and ask which one has production traces:
+
+```bash
+langsmith-cli --json projects list --limit 10 2>/dev/null
+```
+
+Use AskUserQuestion with **preview** (single-select with side-by-side). Build options dynamically from the discovered projects:
+
+```json
+{
+  "questions": [{
+    "question": "LangSmith detected. Which project has your production traces?",
+    "header": "LangSmith",
+    "multiSelect": false,
+    "options": [
+      {
+        "label": "{project_name_1}",
+        "description": "{run_count} runs, last active {date}",
+        "preview": "## {project_name_1}\n\n- **Runs:** {run_count}\n- **Last active:** {date}\n- **Created:** {created_date}\n\nSelecting this project will:\n1. Fetch up to 100 recent traces\n2. Analyze traffic distribution and error patterns\n3. Generate production_seed.md for testgen\n4. Proposers will see real usage data"
+      },
+      {
+        "label": "{project_name_2}",
+        "description": "{run_count} runs, last active {date}",
+        "preview": "## {project_name_2}\n\n- **Runs:** {run_count}\n- **Last active:** {date}\n- **Created:** {created_date}\n\n(same explanation)"
+      },
+      {
+        "label": "Skip",
+        "description": "Don't use production traces",
+        "preview": "## Skip Production Traces\n\nThe evolver will work without production data:\n- Testgen generates synthetic tasks from code analysis\n- No real-world traffic distribution\n- No production error patterns\n\nYou can import traces later with:\n`/harness-evolver:import-traces`"
+      }
+    ]
+  }]
+}
+```
+
+Build the options from the `langsmith-cli` output. Use up to 3 projects (sorted by most recent activity) + the "Skip" option. Fill in actual values for run_count, date, etc.
+
+If a project is selected, pass it as `--langsmith-project` to init.py.
+
 ## Phase 2: Create What's Missing
 
 Three artifacts needed. For each — use existing if found, create if not.
 
 **Harness** (`harness.py`): If user's entry point doesn't match our CLI interface (`--input`, `--output`, `--traces-dir`, `--config`), create a thin wrapper that imports their code. Read their entry point first to understand the I/O format. Ask if unsure.
 
-**Eval** (`eval.py`): If an eval script exists, use it.
+**Eval** (`eval.py`): If an eval script exists, use it. If the user already chose an eval mode in Phase 1.8, follow that choice.
 
-If NO eval exists:
+If NO eval exists and no mode was chosen yet:
 - Copy `eval_passthrough.py` from `$TOOLS/eval_passthrough.py` as the project's eval.py:
   ```bash
   cp $TOOLS/eval_passthrough.py eval.py