harness-evolver 0.1.1 → 0.2.1

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "harness-evolver",
-  "version": "0.1.1",
+  "version": "0.2.1",
   "description": "Meta-Harness-style autonomous harness optimization for Claude Code",
   "author": "Raphael Valdetaro Christi Cordeiro",
   "license": "MIT",

package/skills/harness-evolve-init/SKILL.md

@@ -1,53 +1,198 @@
 ---
 name: harness-evolve-init
-description: "Initialize harness evolution in the current project. Sets up .harness-evolver/ with baseline harness, eval script, and tasks."
-argument-hint: "--harness <path> --eval <path> --tasks <path>"
-allowed-tools: [Read, Write, Bash, Glob]
+description: "Initialize harness evolution in the current project. Scans the codebase, identifies the entry point, and helps create harness wrapper, eval script, and test cases if they don't exist."
+argument-hint: "[directory]"
+allowed-tools: [Read, Write, Edit, Bash, Glob, Grep, Agent]
 ---
 
 # /harness-evolve-init
 
 Initialize the Harness Evolver for this project.
 
-## Arguments
+## Usage
 
-- `--harness <path>` — path to the harness script (any executable, typically Python)
-- `--eval <path>` — path to the evaluation script
-- `--tasks <path>` — path to the tasks directory (JSON files with id, input, expected)
+```
+/harness-evolve-init                    # setup in current directory
+/harness-evolve-init ./my-project       # setup in a specific directory
+```
+
+## Your Job: A 3-Phase Setup Wizard
+
+You are the intelligent layer. The init.py tool is dumb — it takes paths. Your job is to figure out what to pass it, creating files if needed.
 
-## What To Do
+### Phase 1: SCAN the project
 
-Run the init tool:
+Read the project structure to understand what exists:
 
 ```bash
-python3 ~/.harness-evolver/tools/init.py \
-    --harness {harness} \
-    --eval {eval} \
-    --tasks {tasks} \
-    --base-dir .harness-evolver \
-    --harness-config {config if provided, else omit} \
-    --tools-dir ~/.harness-evolver/tools
+find . -maxdepth 3 -type f -name "*.py" | head -30
+ls -la
 ```
 
-If `~/.harness-evolver/tools/init.py` does not exist, check `.harness-evolver/tools/init.py` (local override).
+Look for:
+- **Entry point candidates:** `main.py`, `app.py`, `agent.py`, `graph.py`, `pipeline.py`, `bot.py`, `run.py`, or any file with `if __name__` block
+- **Existing eval/test files:** `eval.py`, `test_*.py`, `score.py`, `judge.py`
+- **Existing test data:** `tasks/`, `tests/`, `data/`, `examples/`, `fixtures/`, any dir with JSON/JSONL files
+- **Config files:** `config.json`, `config.yaml`, `.env`
+- **Framework clues:** imports of langchain, langgraph, openai, anthropic, crewai, etc.
 
-After init completes, report:
-- Baseline score
-- Number of tasks
-- Next step: run `/harness-evolve` to start the optimization loop
+Also run stack detection:
+```bash
+python3 ~/.harness-evolver/tools/detect_stack.py .
+```
+
+### Phase 2: CREATE what's missing
+
+There are 3 artifacts needed. For each, check if it exists or needs to be created.
+
+#### A. The Harness (`harness.py`)
+
+**If `harness.py` already exists with the right interface** (`--input`, `--output`): use it directly.
+
+**If the project has an entry point but NOT in our format:** Create a `harness.py` wrapper.
+
+Read the entry point to understand its input/output format, then generate a wrapper:
+
+```python
+#!/usr/bin/env python3
+"""Harness wrapper for [project name]. Generated by harness-evolve-init."""
+
+import argparse
+import json
+import sys
+
+# Import the actual project code
+from [entry_module] import [main_function_or_class]
+
+def main():
+    parser = argparse.ArgumentParser()
+    parser.add_argument("--input", required=True)
+    parser.add_argument("--output", required=True)
+    parser.add_argument("--traces-dir", default=None)
+    parser.add_argument("--config", default=None)
+    args = parser.parse_args()
+
+    task = json.load(open(args.input))
+    config = json.load(open(args.config)) if args.config else {}
+
+    # Call the actual project code
+    result = [call_the_project](task["input"], **config)
+
+    json.dump({"id": task["id"], "output": str(result)}, open(args.output, "w"))
+
+if __name__ == "__main__":
+    main()
+```
+
+Adapt this template based on what you learned from reading the entry point. Ask the user to confirm if you're unsure about the input/output mapping.
+
+#### B. The Eval (`eval.py`)
+
+**If `eval.py` exists:** use it.
+
+**If not:** Ask the user what "correct" means for their use case, then generate one:
 
-## LangSmith Dataset (optional)
+- **Classification/extraction:** exact match or fuzzy match
+- **Chatbot/QA:** LLM-as-judge (requires API key) or keyword matching
+- **Code generation:** execution-based (run the code, check output)
+- **RAG:** relevance scoring
 
-If the user provides `--langsmith-dataset <dataset_id>`:
+Start with the simplest eval that works. The evolver can iterate on the harness without a perfect eval — even a rough eval gives signal.
+
+```python
+#!/usr/bin/env python3
+"""Eval script for [project]. Generated by harness-evolve-init."""
+
+import argparse
+import json
+import os
+
+def main():
+    parser = argparse.ArgumentParser()
+    parser.add_argument("--results-dir", required=True)
+    parser.add_argument("--tasks-dir", required=True)
+    parser.add_argument("--scores", required=True)
+    args = parser.parse_args()
+
+    correct, total = 0, 0
+    per_task = {}
+
+    for fname in sorted(os.listdir(args.tasks_dir)):
+        if not fname.endswith(".json"):
+            continue
+        with open(os.path.join(args.tasks_dir, fname)) as f:
+            task = json.load(f)
+        task_id = task["id"]
+
+        result_path = os.path.join(args.results_dir, fname)
+        if not os.path.exists(result_path):
+            per_task[task_id] = {"score": 0.0, "error": "no output"}
+            total += 1
+            continue
+
+        with open(result_path) as f:
+            result = json.load(f)
+
+        # ADAPT THIS: define what "correct" means for this project
+        expected = task.get("expected", "")
+        actual = result.get("output", "")
+        match = actual.lower().strip() == expected.lower().strip()
+
+        per_task[task_id] = {"score": 1.0 if match else 0.0, "expected": expected, "actual": actual}
+        correct += int(match)
+        total += 1
+
+    accuracy = correct / total if total > 0 else 0.0
+    json.dump({
+        "combined_score": accuracy,
+        "accuracy": accuracy,
+        "total_tasks": total,
+        "correct": correct,
+        "per_task": per_task,
+    }, open(args.scores, "w"), indent=2)
+
+if __name__ == "__main__":
+    main()
+```
+
+#### C. Test Cases (`tasks/`)
+
+**If `tasks/` exists with JSON files:** use it.
+
+**If the project has test data in another format:** Convert it to our format.
+
+**If no test data exists:** Help create 5-10 test cases. Ask the user:
+> "What are typical inputs to your system? And what are the expected outputs? Give me 3-5 examples and I'll create the task files."
+
+Each task file is:
+```json
+{"id": "task_001", "input": "...", "expected": "...", "metadata": {}}
+```
+
+### Phase 3: RUN init.py
+
+Once all 3 artifacts exist, run:
 
 ```bash
 python3 ~/.harness-evolver/tools/init.py \
-    --harness {harness} \
-    --eval {eval} \
-    --tasks {tasks} \
-    --base-dir .harness-evolver \
-    --langsmith-dataset {dataset_id}
+    --harness harness.py \
+    --eval eval.py \
+    --tasks tasks/ \
+    --tools-dir ~/.harness-evolver/tools
 ```
 
-This pulls examples from a LangSmith dataset to use as tasks.
-Requires `LANGSMITH_API_KEY` in the environment.
+If a harness config exists, add `--harness-config config.json`.
+
+If `~/.harness-evolver/tools/init.py` does not exist, check `.harness-evolver/tools/init.py`.
+
+### After init completes, report:
+
+- What was detected vs created
+- Stack detected (libraries)
+- Integrations available (LangSmith, Context7)
+- Baseline score
+- Next step: `/harness-evolve` to start the optimization loop
+
+## Key Principle
+
+**Don't ask the user to restructure their project.** You adapt to them. If they have a LangGraph graph in `src/graph.py`, you create a thin wrapper — you don't ask them to rename it to `harness.py`. The wrapper IS the harness.

package/tools/init.py

@@ -2,15 +2,17 @@
 """Project initializer for Harness Evolver.
 
 Usage:
-    init.py --harness PATH --eval PATH --tasks PATH --base-dir PATH
-            [--harness-config PATH] [--tools-dir PATH]
+    init.py [DIR]                                          # auto-detect in DIR (or CWD)
+    init.py --harness PATH --eval PATH --tasks PATH        # explicit paths
+    init.py --base-dir PATH [--harness-config PATH]        # advanced options
 
-Creates the .harness-evolver/ directory structure, copies baseline files,
-runs validation, evaluates the baseline, and initializes state.
+Auto-detects harness.py, eval.py, tasks/ and config.json in the working directory.
+Falls back to fuzzy matching (*harness*, *eval*, *score*, dirs with .json files).
 Stdlib-only. No external dependencies.
 """
 
 import argparse
+import glob
 import json
 import os
 import shutil
@@ -19,6 +21,74 @@ import sys
 import tempfile
 
 
+def _auto_detect(search_dir):
+    """Auto-detect harness, eval, and tasks in a directory.
+
+    Returns (harness_path, eval_path, tasks_path, config_path) or raises SystemExit.
+    """
+    search_dir = os.path.abspath(search_dir)
+
+    # Exact convention names first
+    harness = None
+    eval_script = None
+    tasks = None
+    config = None
+
+    # 1. Exact matches
+    for name in ["harness.py"]:
+        p = os.path.join(search_dir, name)
+        if os.path.isfile(p):
+            harness = p
+    for name in ["eval.py"]:
+        p = os.path.join(search_dir, name)
+        if os.path.isfile(p):
+            eval_script = p
+    for name in ["tasks", "tasks/"]:
+        p = os.path.join(search_dir, name.rstrip("/"))
+        if os.path.isdir(p):
+            tasks = p
+    for name in ["config.json"]:
+        p = os.path.join(search_dir, name)
+        if os.path.isfile(p):
+            config = p
+
+    # 2. Fuzzy fallback for harness
+    if not harness:
+        candidates = [f for f in glob.glob(os.path.join(search_dir, "*.py"))
+                      if any(k in os.path.basename(f).lower() for k in ["harness", "agent", "run"])]
+        if len(candidates) == 1:
+            harness = candidates[0]
+
+    # 3. Fuzzy fallback for eval
+    if not eval_script:
+        candidates = [f for f in glob.glob(os.path.join(search_dir, "*.py"))
+                      if any(k in os.path.basename(f).lower() for k in ["eval", "score", "judge"])
+                      and f != harness]
+        if len(candidates) == 1:
+            eval_script = candidates[0]
+
+    # 4. Fuzzy fallback for tasks
+    if not tasks:
+        for d in os.listdir(search_dir):
+            dp = os.path.join(search_dir, d)
+            if os.path.isdir(dp) and any(f.endswith(".json") for f in os.listdir(dp)):
+                # Check if at least one JSON has "id" and "input" keys
+                for f in os.listdir(dp):
+                    if f.endswith(".json"):
+                        try:
+                            with open(os.path.join(dp, f)) as fh:
+                                data = json.load(fh)
+                            if "id" in data and "input" in data:
+                                tasks = dp
+                                break
+                        except (json.JSONDecodeError, KeyError):
+                            pass
+                if tasks:
+                    break
+
+    return harness, eval_script, tasks, config
+
+
 def _detect_langsmith():
     """Auto-detect LangSmith API key and return config section."""
     if os.environ.get("LANGSMITH_API_KEY"):
@@ -77,16 +147,59 @@ def _check_context7_available():
 
 
 def main():
-    parser = argparse.ArgumentParser(description="Initialize Harness Evolver project")
-    parser.add_argument("--harness", required=True, help="Path to harness script")
-    parser.add_argument("--eval", required=True, help="Path to eval script")
-    parser.add_argument("--tasks", required=True, help="Path to tasks directory")
-    parser.add_argument("--base-dir", required=True, help="Path for .harness-evolver/")
+    parser = argparse.ArgumentParser(
+        description="Initialize Harness Evolver project",
+        usage="init.py [DIR] [--harness PATH] [--eval PATH] [--tasks PATH]",
+    )
+    parser.add_argument("dir", nargs="?", default=".",
+                        help="Directory to scan (default: current directory)")
+    parser.add_argument("--harness", default=None, help="Path to harness script")
+    parser.add_argument("--eval", default=None, help="Path to eval script")
+    parser.add_argument("--tasks", default=None, help="Path to tasks directory")
+    parser.add_argument("--base-dir", default=None, help="Path for .harness-evolver/")
     parser.add_argument("--harness-config", default=None, help="Path to harness config.json")
     parser.add_argument("--tools-dir", default=None, help="Path to tools directory")
     args = parser.parse_args()
 
-    base = args.base_dir
+    # Auto-detect missing args
+    search_dir = os.path.abspath(args.dir)
+    if not args.harness or not args.eval or not args.tasks:
+        detected_harness, detected_eval, detected_tasks, detected_config = _auto_detect(search_dir)
+        if not args.harness:
+            args.harness = detected_harness
+        if not args.eval:
+            args.eval = detected_eval
+        if not args.tasks:
+            args.tasks = detected_tasks
+        if not args.harness_config and detected_config:
+            args.harness_config = detected_config
+
+    # Validate we have everything
+    missing = []
+    if not args.harness:
+        missing.append("harness (no harness.py or *harness*.py found)")
+    if not args.eval:
+        missing.append("eval (no eval.py or *eval*.py found)")
+    if not args.tasks:
+        missing.append("tasks (no tasks/ directory with JSON files found)")
+    if missing:
+        print("Could not auto-detect:", file=sys.stderr)
+        for m in missing:
+            print(f"  - {m}", file=sys.stderr)
+        print(f"\nSearched in: {search_dir}", file=sys.stderr)
+        print("\nProvide explicitly:", file=sys.stderr)
+        print("  /harness-evolve-init --harness PATH --eval PATH --tasks PATH", file=sys.stderr)
+        sys.exit(1)
+
+    # Print what was detected
+    print(f"Harness: {os.path.relpath(args.harness, search_dir)}")
+    print(f"Eval:    {os.path.relpath(args.eval, search_dir)}")
+    print(f"Tasks:   {os.path.relpath(args.tasks, search_dir)}/")
+    if args.harness_config:
+        print(f"Config:  {os.path.relpath(args.harness_config, search_dir)}")
+    print()
+
+    base = args.base_dir or os.path.join(search_dir, ".harness-evolver")
     tools = args.tools_dir or os.path.dirname(__file__)
 
     evaluate_py = os.path.join(tools, "evaluate.py")