harness-evolver 0.2.0 → 0.2.1

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "harness-evolver",
-  "version": "0.2.0",
+  "version": "0.2.1",
   "description": "Meta-Harness-style autonomous harness optimization for Claude Code",
   "author": "Raphael Valdetaro Christi Cordeiro",
   "license": "MIT",

package/skills/harness-evolve-init/SKILL.md

@@ -1,8 +1,8 @@
 ---
 name: harness-evolve-init
-description: "Initialize harness evolution in the current project. Auto-detects harness.py, eval.py, and tasks/ in the working directory."
-argument-hint: "[directory] [--harness <path>] [--eval <path>] [--tasks <path>]"
-allowed-tools: [Read, Write, Bash, Glob]
+description: "Initialize harness evolution in the current project. Scans the codebase, identifies the entry point, and helps create harness wrapper, eval script, and test cases if they don't exist."
+argument-hint: "[directory]"
+allowed-tools: [Read, Write, Edit, Bash, Glob, Grep, Agent]
 ---
 
 # /harness-evolve-init
@@ -12,39 +12,187 @@ Initialize the Harness Evolver for this project.
 ## Usage
 
 ```
-/harness-evolve-init                    # auto-detect everything in CWD
-/harness-evolve-init ./my-project       # auto-detect in a specific directory
-/harness-evolve-init --harness run.py   # override one path, auto-detect the rest
+/harness-evolve-init                    # setup in current directory
+/harness-evolve-init ./my-project       # setup in a specific directory
 ```
 
-## How Auto-Detection Works
+## Your Job: A 3-Phase Setup Wizard
 
-The tool scans the directory for:
-1. **Exact names:** `harness.py`, `eval.py`, `tasks/`, `config.json`
-2. **Fuzzy fallback:** `*harness*`, `*agent*`, `*run*` for harness; `*eval*`, `*score*` for eval; any dir with JSON files containing `id`/`input` fields for tasks
+You are the intelligent layer. The init.py tool is dumb — it takes paths. Your job is to figure out what to pass it, creating files if needed.
 
-If all 3 are found, init proceeds immediately. If something is missing, it reports what's needed.
+### Phase 1: SCAN the project
 
-## What To Do
+Read the project structure to understand what exists:
 
-Run the init tool:
+```bash
+find . -maxdepth 3 -type f -name "*.py" | head -30
+ls -la
+```
+
+Look for:
+- **Entry point candidates:** `main.py`, `app.py`, `agent.py`, `graph.py`, `pipeline.py`, `bot.py`, `run.py`, or any file with `if __name__` block
+- **Existing eval/test files:** `eval.py`, `test_*.py`, `score.py`, `judge.py`
+- **Existing test data:** `tasks/`, `tests/`, `data/`, `examples/`, `fixtures/`, any dir with JSON/JSONL files
+- **Config files:** `config.json`, `config.yaml`, `.env`
+- **Framework clues:** imports of langchain, langgraph, openai, anthropic, crewai, etc.
+
+Also run stack detection:
+```bash
+python3 ~/.harness-evolver/tools/detect_stack.py .
+```
+
+### Phase 2: CREATE what's missing
+
+There are 3 artifacts needed. For each, check if it exists or needs to be created.
+
+#### A. The Harness (`harness.py`)
+
+**If `harness.py` already exists with the right interface** (`--input`, `--output`): use it directly.
+
+**If the project has an entry point but NOT in our format:** Create a `harness.py` wrapper.
+
+Read the entry point to understand its input/output format, then generate a wrapper:
+
+```python
+#!/usr/bin/env python3
+"""Harness wrapper for [project name]. Generated by harness-evolve-init."""
+
+import argparse
+import json
+import sys
+
+# Import the actual project code
+from [entry_module] import [main_function_or_class]
+
+def main():
+    parser = argparse.ArgumentParser()
+    parser.add_argument("--input", required=True)
+    parser.add_argument("--output", required=True)
+    parser.add_argument("--traces-dir", default=None)
+    parser.add_argument("--config", default=None)
+    args = parser.parse_args()
+
+    task = json.load(open(args.input))
+    config = json.load(open(args.config)) if args.config else {}
+
+    # Call the actual project code
+    result = [call_the_project](task["input"], **config)
+
+    json.dump({"id": task["id"], "output": str(result)}, open(args.output, "w"))
+
+if __name__ == "__main__":
+    main()
+```
+
+Adapt this template based on what you learned from reading the entry point. Ask the user to confirm if you're unsure about the input/output mapping.
+
+#### B. The Eval (`eval.py`)
+
+**If `eval.py` exists:** use it.
+
+**If not:** Ask the user what "correct" means for their use case, then generate one:
+
+- **Classification/extraction:** exact match or fuzzy match
+- **Chatbot/QA:** LLM-as-judge (requires API key) or keyword matching
+- **Code generation:** execution-based (run the code, check output)
+- **RAG:** relevance scoring
+
+Start with the simplest eval that works. The evolver can iterate on the harness without a perfect eval — even a rough eval gives signal.
+
+```python
+#!/usr/bin/env python3
+"""Eval script for [project]. Generated by harness-evolve-init."""
+
+import argparse
+import json
+import os
+
+def main():
+    parser = argparse.ArgumentParser()
+    parser.add_argument("--results-dir", required=True)
+    parser.add_argument("--tasks-dir", required=True)
+    parser.add_argument("--scores", required=True)
+    args = parser.parse_args()
+
+    correct, total = 0, 0
+    per_task = {}
+
+    for fname in sorted(os.listdir(args.tasks_dir)):
+        if not fname.endswith(".json"):
+            continue
+        with open(os.path.join(args.tasks_dir, fname)) as f:
+            task = json.load(f)
+        task_id = task["id"]
+
+        result_path = os.path.join(args.results_dir, fname)
+        if not os.path.exists(result_path):
+            per_task[task_id] = {"score": 0.0, "error": "no output"}
+            total += 1
+            continue
+
+        with open(result_path) as f:
+            result = json.load(f)
+
+        # ADAPT THIS: define what "correct" means for this project
+        expected = task.get("expected", "")
+        actual = result.get("output", "")
+        match = actual.lower().strip() == expected.lower().strip()
+
+        per_task[task_id] = {"score": 1.0 if match else 0.0, "expected": expected, "actual": actual}
+        correct += int(match)
+        total += 1
+
+    accuracy = correct / total if total > 0 else 0.0
+    json.dump({
+        "combined_score": accuracy,
+        "accuracy": accuracy,
+        "total_tasks": total,
+        "correct": correct,
+        "per_task": per_task,
+    }, open(args.scores, "w"), indent=2)
+
+if __name__ == "__main__":
+    main()
+```
+
+#### C. Test Cases (`tasks/`)
+
+**If `tasks/` exists with JSON files:** use it.
+
+**If the project has test data in another format:** Convert it to our format.
+
+**If no test data exists:** Help create 5-10 test cases. Ask the user:
+> "What are typical inputs to your system? And what are the expected outputs? Give me 3-5 examples and I'll create the task files."
+
+Each task file is:
+```json
+{"id": "task_001", "input": "...", "expected": "...", "metadata": {}}
+```
+
+### Phase 3: RUN init.py
+
+Once all 3 artifacts exist, run:
 
 ```bash
-python3 ~/.harness-evolver/tools/init.py {directory if provided} \
+python3 ~/.harness-evolver/tools/init.py \
+    --harness harness.py \
+    --eval eval.py \
+    --tasks tasks/ \
     --tools-dir ~/.harness-evolver/tools
 ```
 
-Add explicit flags only if the user provided them:
-- `--harness PATH` — override harness auto-detection
-- `--eval PATH` — override eval auto-detection
-- `--tasks PATH` — override tasks auto-detection
-- `--harness-config PATH` — optional config for the harness
+If a harness config exists, add `--harness-config config.json`.
 
-If `~/.harness-evolver/tools/init.py` does not exist, check `.harness-evolver/tools/init.py` (local override).
+If `~/.harness-evolver/tools/init.py` does not exist, check `.harness-evolver/tools/init.py`.
 
-After init completes, report:
-- What was detected (harness, eval, tasks)
+### After init completes, report:
+
+- What was detected vs created
+- Stack detected (libraries)
+- Integrations available (LangSmith, Context7)
 - Baseline score
-- Number of tasks
-- Integrations detected (LangSmith, Context7, stack)
-- Next step: run `/harness-evolve` to start the optimization loop
+- Next step: `/harness-evolve` to start the optimization loop
+
+## Key Principle
+
+**Don't ask the user to restructure their project.** You adapt to them. If they have a LangGraph graph in `src/graph.py`, you create a thin wrapper — you don't ask them to rename it to `harness.py`. The wrapper IS the harness.