harness-evolver 3.0.5 → 3.1.0

package/README.md

@@ -47,7 +47,7 @@ claude
 <table>
 <tr>
 <td><b>LangSmith-Native</b></td>
-<td>No custom eval scripts or task files. Uses LangSmith Datasets for test inputs, Experiments for results, and Evaluators (openevals LLM-as-judge) for scoring. Everything is visible in the LangSmith UI.</td>
+<td>No custom eval scripts or task files. Uses LangSmith Datasets for test inputs, Experiments for results, and an agent-based LLM-as-judge for scoring via langsmith-cli. No external API keys needed. Everything is visible in the LangSmith UI.</td>
 </tr>
 <tr>
 <td><b>Real Code Evolution</b></td>
@@ -92,6 +92,7 @@ claude
 | **Architect** | Recommends multi-agent topology changes | Blue |
 | **Critic** | Validates evaluator quality, detects gaming | Red |
 | **TestGen** | Generates test inputs for LangSmith datasets | Cyan |
+| **Evaluator** | LLM-as-judge — reads outputs via langsmith-cli, scores correctness | Yellow |
 
 ---
 
@@ -104,7 +105,8 @@ claude
   +- 1.5 Gather trace insights (cluster errors, tokens, latency)
   +- 1.8 Analyze per-task failures (adaptive briefings)
   +- 2.  Spawn 5 proposers in parallel (each in a git worktree)
-  +- 3.  Evaluate each candidate (client.evaluate() -> LangSmith experiments)
+  +- 3.  Run target for each candidate (client.evaluate() -> code-based evaluators)
+  +- 3.5 Spawn evaluator agent (reads outputs via langsmith-cli, judges, writes scores)
   +- 4.  Compare experiments -> select winner + per-task champion
   +- 5.  Merge winning worktree into main branch
   +- 5.5 Test suite growth (add regression examples to dataset)
@@ -119,13 +121,15 @@ claude
 ## Requirements
 
 - **LangSmith account** + `LANGSMITH_API_KEY`
-- **Python 3.10+** with `langsmith` and `openevals` packages
+- **Python 3.10+** with `langsmith` package
+- **langsmith-cli** (`uv tool install langsmith-cli`) — required for evaluator agent
 - **Git** (for worktree-based isolation)
 - **Claude Code** (or Cursor/Codex/Windsurf)
 
 ```bash
 export LANGSMITH_API_KEY="lsv2_pt_..."
-pip install langsmith openevals
+pip install langsmith
+uv tool install langsmith-cli
 ```
 
 ---

package/agents/evolver-evaluator.md

@@ -0,0 +1,152 @@
+---
+name: evolver-evaluator
+description: |
+  Use this agent to evaluate experiment outputs using LLM-as-judge.
+  Reads run inputs/outputs from LangSmith via langsmith-cli, judges correctness,
+  and writes scores back as feedback. No external API keys needed.
+tools: Read, Bash, Glob, Grep
+color: yellow
+---
+
+# Evolver — Evaluator Agent (v3)
+
+You are an LLM evaluation judge. Your job is to read the outputs of an experiment from LangSmith, evaluate each one for correctness, and write scores back as feedback.
+
+You ARE the LLM-as-judge. You replace the need for an external LLM API call.
+
+## Bootstrap
+
+1. Verify langsmith-cli is available:
+```bash
+langsmith-cli --version
+```
+If this fails, report the error and stop — langsmith-cli is required.
+
+2. Your prompt contains `<experiment>`, `<evaluators>`, and `<context>` blocks. Parse them to understand:
+- Which experiment to evaluate
+- What evaluation criteria to apply
+- What the agent is supposed to do (domain context)
+
+## Tool: langsmith-cli
+
+You interact with LangSmith exclusively through `langsmith-cli`. Always use `--json` for machine-readable output.
+
+### Reading experiment outputs
+
+```bash
+langsmith-cli --json runs list \
+    --project "{experiment_name}" \
+    --fields id,inputs,outputs,error,reference_example_id \
+    --is-root \
+    --limit 200
+```
+
+This returns one JSON object per line (JSONL). Each line has:
+- `id` — the run ID (needed to write feedback)
+- `inputs` — what was sent to the agent
+- `outputs` — what the agent responded
+- `error` — error message if the run failed
+- `reference_example_id` — links back to the dataset example
+
+### Writing scores
+
+For EACH run, after judging it:
+
+```bash
+langsmith-cli --json feedback create {run_id} \
+    --key "{evaluator_key}" \
+    --score {score} \
+    --comment "{brief_reasoning}" \
+    --source model
+```
+
+Use `--source model` since this is an LLM-generated evaluation.
+
+## Your Workflow
+
+### Phase 1: Read All Outputs
+
+Fetch all runs from the experiment. Save the output to a file for reference:
+
+```bash
+langsmith-cli --json runs list \
+    --project "{experiment_name}" \
+    --fields id,inputs,outputs,error,reference_example_id \
+    --is-root --limit 200 \
+    --output experiment_runs.jsonl
+```
+
+Then read `experiment_runs.jsonl` to see all results.
+
+### Phase 2: Evaluate Each Run
+
+For each run, apply the requested evaluators. The evaluators you may be asked to judge:
+
+#### correctness
+Judge: **Is the output a correct, accurate, and complete response to the input?**
+
+Scoring:
+- `1.0` — Correct and complete. The response accurately addresses the input.
+- `0.0` — Incorrect, incomplete, or off-topic.
+
+Consider:
+- Does the response answer what was asked?
+- Is the information factually accurate?
+- Are there hallucinations or made-up facts?
+- Is the response relevant to the domain?
+
+#### conciseness
+Judge: **Is the response appropriately concise without sacrificing quality?**
+
+Scoring:
+- `1.0` — Concise and complete. No unnecessary verbosity.
+- `0.0` — Excessively verbose, repetitive, or padded.
+
+### Phase 3: Write All Scores
+
+For each run you evaluated, write feedback via `langsmith-cli feedback create`.
+
+Write scores in batches — evaluate all runs first, then write all scores. This is more efficient than alternating between reading and writing.
+
+Example for one run:
+```bash
+langsmith-cli --json feedback create "run-uuid-here" \
+    --key correctness \
+    --score 1.0 \
+    --comment "Response correctly identifies the applicable regulation and provides accurate guidance." \
+    --source model
+```
+
+### Phase 4: Summary
+
+After writing all scores, compute the aggregate:
+
+```bash
+langsmith-cli --json feedback list --run-id "{any_run_id}" --key correctness
+```
+
+## Error Handling
+
+- If a run has `error` set and empty `outputs`: score it `0.0` with comment "Run failed: {error}"
+- If a run has `outputs` but they contain an error message: score `0.0` with comment explaining the failure
+- If `outputs` is empty but no error: score `0.0` with comment "Empty output"
+
+## Rules
+
+1. **Be a fair judge** — evaluate based on the criteria, not your preferences
+2. **Brief comments** — keep feedback comments under 200 characters
+3. **Binary scoring for correctness** — use 1.0 or 0.0, not partial scores (unless instructed otherwise)
+4. **Score EVERY run** — don't skip any, even failed ones
+5. **Domain awareness** — use the `<context>` block to understand what constitutes a "correct" answer in this domain
+
+## Return Protocol
+
+When done, end your response with:
+
+## EVALUATION COMPLETE
+- **Experiment**: {experiment_name}
+- **Runs evaluated**: {N}
+- **Evaluators applied**: {list}
+- **Mean score**: {score}
+- **Pass rate**: {N}/{total} ({percent}%)
+- **Common failure patterns**: {brief list}

package/agents/evolver-proposer.md

@@ -64,7 +64,34 @@ Based on your strategy and diagnosis, modify the code:
 - **Error handling**: retry logic, fallback strategies, timeout handling
 - **Model selection**: which model for which task
 
-Use Context7 MCP tools (`resolve-library-id`, `get-library-docs`) to check current API documentation before writing code that uses library APIs.
+### Phase 3.5: Consult Documentation (MANDATORY)
+
+**Before writing ANY code**, you MUST consult Context7 for every library you'll be modifying or using. This is NOT optional.
+
+**Step 1 — Identify libraries from the code you read:**
+Read the imports in the files you're about to modify. For each framework/library (LangGraph, OpenAI, Anthropic, CrewAI, etc.):
+
+**Step 2 — Resolve library ID:**
+```
+resolve-library-id(libraryName: "langgraph", query: "what you're trying to do")
+```
+This returns up to 10 matches. Pick the one with the highest relevance.
+
+**Step 3 — Query docs for your specific task:**
+```
+get-library-docs(libraryId: "/langchain-ai/langgraph", query: "conditional edges StateGraph", topic: "routing")
+```
+Ask about the SPECIFIC API you're going to use or change.
+
+**Examples of what to query:**
+- About to modify a StateGraph? → `query: "StateGraph add_conditional_edges"` 
+- Changing prompt template? → `query: "ChatPromptTemplate from_messages"` for langchain
+- Adding a tool? → `query: "StructuredTool create tool definition"` for langchain
+- Changing model? → `query: "ChatOpenAI model parameters temperature"` for openai
+
+**Why this matters:** Your training data may be outdated. Libraries change APIs between versions. A quick Context7 lookup takes seconds and prevents proposing code that uses deprecated or incorrect patterns. The documentation is the source of truth, not your model knowledge.
+
+**If Context7 MCP is not available:** Note in proposal.md "API patterns not verified against current docs — verify before deploying."
 
 ### Phase 4: Commit and Document
 
@@ -99,15 +126,6 @@ If `production_seed.json` exists:
 
 Prioritize changes that fix real production failures over synthetic test failures.
 
-## Context7 — Documentation Lookup
-
-Use Context7 MCP tools proactively when:
-- Writing code that uses a library API
-- Unsure about method signatures or patterns
-- Checking if a better approach exists in the latest version
-
-If Context7 is not available, proceed with model knowledge but note in proposal.md.
-
 ## Rules
 
 1. **Read before writing** — understand the code before changing it

package/bin/install.js

@@ -2,7 +2,7 @@
 /**
  * Harness Evolver v3 installer.
  * Copies skills/agents/tools to runtime directories (GSD pattern).
- * Installs Python dependencies (langsmith + openevals).
+ * Installs Python dependencies (langsmith) and langsmith-cli.
  *
  * Usage: npx harness-evolver@latest
  */
@@ -225,15 +225,15 @@ function installPythonDeps() {
 
   // Install/upgrade deps in the venv
   const installCommands = [
-    `uv pip install --python "${venvPython}" langsmith openevals`,
-    `"${venvPip}" install --upgrade langsmith openevals`,
-    `"${venvPython}" -m pip install --upgrade langsmith openevals`,
+    `uv pip install --python "${venvPython}" langsmith`,
+    `"${venvPip}" install --upgrade langsmith`,
+    `"${venvPython}" -m pip install --upgrade langsmith`,
   ];
 
   for (const cmd of installCommands) {
     try {
       execSync(cmd, { stdio: "pipe", timeout: 120000 });
-      console.log(`  ${GREEN}✓${RESET} langsmith + openevals installed in venv`);
+      console.log(`  ${GREEN}✓${RESET} langsmith installed in venv`);
       return true;
     } catch {
       continue;
@@ -241,7 +241,7 @@ function installPythonDeps() {
   }
 
   console.log(`  ${YELLOW}!${RESET} Could not install packages in venv.`);
-  console.log(`    Run manually: ${BOLD}~/.evolver/venv/bin/pip install langsmith openevals${RESET}`);
+  console.log(`    Run manually: ${BOLD}~/.evolver/venv/bin/pip install langsmith${RESET}`);
   return false;
 }
 
@@ -303,26 +303,24 @@ async function configureLangSmith(rl) {
     }
   }
 
-  // --- Step 2: langsmith-cli ---
+  // --- Step 2: langsmith-cli (required for evaluator agent) ---
   if (hasLangsmithCli) {
     console.log(`  ${GREEN}✓${RESET} langsmith-cli installed`);
   } else {
-    console.log(`\n  ${BOLD}langsmith-cli${RESET} — optional but useful for debugging traces`);
-    console.log(`  ${DIM}Quick project listing, trace inspection, run stats from terminal.${RESET}`);
-    const lsCliAnswer = await ask(rl, `\n  ${YELLOW}Install langsmith-cli? [Y/n]:${RESET} `);
-    if (lsCliAnswer.trim().toLowerCase() !== "n") {
-      console.log(`\n  Installing langsmith-cli...`);
-      try {
-        execSync("uv tool install langsmith-cli 2>/dev/null || pip install langsmith-cli 2>/dev/null || pip3 install langsmith-cli", { stdio: "pipe", timeout: 60000 });
-        console.log(`  ${GREEN}✓${RESET} langsmith-cli installed`);
+    console.log(`\n  ${BOLD}langsmith-cli${RESET} — ${YELLOW}required${RESET} for LLM-as-judge evaluation`);
+    console.log(`  ${DIM}The evaluator agent uses it to read experiment outputs and write scores.${RESET}`);
+    console.log(`\n  Installing langsmith-cli...`);
+    try {
+      execSync("uv tool install langsmith-cli 2>/dev/null || pip install langsmith-cli 2>/dev/null || pip3 install langsmith-cli", { stdio: "pipe", timeout: 60000 });
+      console.log(`  ${GREEN}✓${RESET} langsmith-cli installed`);
 
-        // If we have a key, auto-authenticate
-        if (hasKey && fs.existsSync(langsmithCredsFile)) {
-          console.log(`  ${GREEN}✓${RESET} langsmith-cli auto-authenticated (credentials file exists)`);
-        }
-      } catch {
-        console.log(`  ${YELLOW}!${RESET} Could not install. Try manually: ${DIM}uv tool install langsmith-cli${RESET}`);
+      // If we have a key, auto-authenticate
+      if (hasKey && fs.existsSync(langsmithCredsFile)) {
+        console.log(`  ${GREEN}✓${RESET} langsmith-cli auto-authenticated (credentials file exists)`);
       }
+    } catch {
+      console.log(`  ${RED}!${RESET} Could not install langsmith-cli.`);
+      console.log(`    ${BOLD}This is required.${RESET} Install manually: ${DIM}uv tool install langsmith-cli${RESET}`);
     }
   }
 }

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "harness-evolver",
-  "version": "3.0.5",
+  "version": "3.1.0",
   "description": "LangSmith-native autonomous agent optimization for Claude Code",
   "author": "Raphael Valdetaro",
   "license": "MIT",

package/skills/evolve/SKILL.md

@@ -172,7 +172,7 @@ If ALL_PASSING: D gets `creative`, E gets `efficiency`.
 
 Wait for all 5 to complete.
 
-### 3. Evaluate Each Candidate
+### 3. Run Target for Each Candidate
 
 For each worktree that has changes (proposer committed something):
 
@@ -184,7 +184,59 @@ $EVOLVER_PY $TOOLS/run_eval.py \
     --timeout 120
 ```
 
-Each candidate becomes a separate LangSmith experiment.
+Each candidate becomes a separate LangSmith experiment. This step runs the agent and applies code-based evaluators (has_output, token_efficiency) only.
+
+Collect all experiment names from the output (the `"experiment"` field in each JSON output).
+
+### 3.5. LLM-as-Judge Evaluation (Evaluator Agent)
+
+Check if the config has LLM-based evaluators (correctness, conciseness):
+
+```bash
+python3 -c "import json; c=json.load(open('.evolver.json')); llm=[k for k in c['evaluators'] if k in ('correctness','conciseness')]; print(','.join(llm) if llm else '')"
+```
+
+If LLM evaluators are configured, first verify langsmith-cli is available:
+
+```bash
+command -v langsmith-cli >/dev/null 2>&1 || { echo "ERROR: langsmith-cli not found. Install with: uv tool install langsmith-cli"; exit 1; }
+```
+
+Then spawn ONE evaluator agent that scores ALL candidates in a single pass. This is more efficient than spawning one agent per candidate:
+
+```
+Agent(
+  subagent_type: "evolver-evaluator",
+  description: "Evaluate all candidates for iteration v{NNN}",
+  prompt: |
+    <experiment>
+    Evaluate the following experiments (one per candidate):
+    - {experiment_name_a}
+    - {experiment_name_b}
+    - {experiment_name_c}
+    - {experiment_name_d}
+    - {experiment_name_e}
+    </experiment>
+
+    <evaluators>
+    Apply these evaluators to each run in each experiment:
+    - {llm_evaluator_list, e.g. "correctness", "conciseness"}
+    </evaluators>
+
+    <context>
+    Agent type: {framework} agent
+    Domain: {description from .evolver.json or entry point context}
+    Entry point: {entry_point}
+
+    For each experiment:
+    1. Read all runs via: langsmith-cli --json runs list --project "{experiment_name}" --fields id,inputs,outputs,error --is-root --limit 200
+    2. Judge each run's output against the input
+    3. Write scores via: langsmith-cli --json feedback create {run_id} --key {evaluator} --score {0.0|1.0} --comment "{reason}" --source model
+    </context>
+)
+```
+
+Wait for the evaluator agent to complete before proceeding.
 
 ### 4. Compare All Candidates
 

package/skills/setup/SKILL.md

@@ -42,25 +42,27 @@ TOOLS=$([ -d ".evolver/tools" ] && echo ".evolver/tools" || echo "$HOME/.evolver
 EVOLVER_PY=$([ -f "$HOME/.evolver/venv/bin/python" ] && echo "$HOME/.evolver/venv/bin/python" || echo "python3")
 ```
 
-Use `$EVOLVER_PY` instead of `python3` for ALL tool invocations. This ensures the venv with langsmith+openevals is used.
+Use `$EVOLVER_PY` instead of `python3` for ALL tool invocations. This ensures the venv with langsmith is used.
 
 ## Phase 1: Explore Project (automatic)
 
 ```bash
-find . -maxdepth 3 -type f -name "*.py" | head -30
-$EVOLVER_PY $TOOLS/detect_stack.py .
+find . -maxdepth 3 -type f -name "*.py" -not -path "*/.venv/*" -not -path "*/node_modules/*" -not -path "*/__pycache__/*" | head -30
 ```
 
+**Monorepo detection**: if the project root has multiple subdirectories with their own `main.py` or `pyproject.toml`, it's a monorepo. Use AskUserQuestion to ask WHICH app to optimize before proceeding — do NOT scan everything.
+
 Look for:
 - Entry points: files with `if __name__`, or named `main.py`, `app.py`, `agent.py`, `graph.py`, `pipeline.py`
-- Framework: LangGraph, CrewAI, OpenAI SDK, Anthropic SDK, etc.
 - Existing LangSmith config: `LANGCHAIN_PROJECT` / `LANGSMITH_PROJECT` in env or `.env`
 - Existing test data: JSON files with inputs, CSV files, etc.
 - Dependencies: `requirements.txt`, `pyproject.toml`
 
-Identify the **run command** — how to execute the agent. Examples:
+To identify the **framework**, read the entry point file and its immediate imports. The proposer agents will use Context7 MCP for detailed documentation lookup — you don't need to detect every library, just identify the main framework (LangGraph, CrewAI, OpenAI Agents SDK, etc.) from the imports you see.
+
+Identify the **run command** — how to execute the agent:
 - `python main.py` (if it accepts `--input` flag)
-- `python -c "from agent import run; import json,sys; print(json.dumps(run(json.load(open(sys.argv[1])))))"`
+- The command in the project's README, Makefile, or scripts/
 
 ## Phase 2: Confirm Detection (interactive)
 
@@ -199,4 +201,4 @@ Next: run /evolver:evolve to start optimizing.
 - If `.evolver.json` already exists, ask before overwriting.
 - If the agent needs a venv, the run command should activate it: `cd {dir} && .venv/bin/python main.py`
 - If LangSmith connection fails, check API key and network.
-- The setup installs `langsmith` and `openevals` if missing.
+- The setup requires `langsmith` (Python SDK) and `langsmith-cli` (for evaluator agent).

package/tools/run_eval.py

@@ -2,7 +2,9 @@
 """Run LangSmith evaluation for a candidate in a worktree.
 
 Wraps client.evaluate() — runs the user's agent against the dataset
-with configured evaluators, from within a specific directory (worktree).
+with code-based evaluators only (has_output, token_efficiency).
+LLM-as-judge scoring (correctness, conciseness) is handled post-hoc
+by the evolver-evaluator agent via langsmith-cli.
 
 Usage:
     python3 run_eval.py \
@@ -11,7 +13,7 @@ Usage:
         --experiment-prefix v001a \
         [--timeout 120]
 
-Requires: pip install langsmith openevals
+Requires: pip install langsmith
 """
 
 import argparse
@@ -124,34 +126,30 @@ def make_target(entry_point, cwd):
 
 
 def load_evaluators(evaluator_keys):
-    """Load evaluators by key name."""
-    from openevals.llm import create_llm_as_judge
-    from openevals.prompts import CORRECTNESS_PROMPT, CONCISENESS_PROMPT
+    """Load code-based evaluators only.
 
+    LLM-as-judge evaluators (correctness, conciseness) are handled
+    post-hoc by the evolver-evaluator agent via langsmith-cli.
+    """
     evaluators = []
+
+    # Always include has_output — verifies the agent produced something
+    def has_output_eval(inputs, outputs, **kwargs):
+        has = bool(outputs and outputs.get("output", outputs.get("answer", "")))
+        return {"key": "has_output", "score": 1.0 if has else 0.0}
+    evaluators.append(has_output_eval)
+
     for key in evaluator_keys:
-        if key == "correctness":
-            evaluators.append(create_llm_as_judge(
-                prompt=CORRECTNESS_PROMPT,
-                feedback_key="correctness",
-                model="openai:gpt-4.1-mini",
-            ))
-        elif key == "conciseness":
-            evaluators.append(create_llm_as_judge(
-                prompt=CONCISENESS_PROMPT,
-                feedback_key="conciseness",
-                model="openai:gpt-4.1-mini",
-            ))
-        elif key == "latency":
-            def latency_eval(inputs, outputs, **kwargs):
-                return {"key": "has_output", "score": 1.0 if outputs else 0.0}
-            evaluators.append(latency_eval)
+        if key == "latency":
+            # Latency is captured in traces, just check output exists
+            pass  # has_output already covers this
         elif key == "token_efficiency":
             def token_eval(inputs, outputs, **kwargs):
                 output_text = str(outputs.get("output", outputs.get("answer", "")))
                 score = min(1.0, 2000 / max(len(output_text), 1))
                 return {"key": "token_efficiency", "score": score}
             evaluators.append(token_eval)
+        # correctness, conciseness — skipped, handled by evaluator agent
 
     return evaluators
 
@@ -176,10 +174,16 @@ def main():
     target = make_target(config["entry_point"], args.worktree_path)
     evaluators = load_evaluators(config["evaluators"])
 
+    # Identify which evaluators need the agent (LLM-as-judge)
+    llm_evaluators = [k for k in config["evaluators"] if k in ("correctness", "conciseness")]
+    code_evaluators = [k for k in config["evaluators"] if k not in ("correctness", "conciseness")]
+
     print(f"Running evaluation: {args.experiment_prefix}")
     print(f"  Dataset: {config['dataset']}")
     print(f"  Worktree: {args.worktree_path}")
-    print(f"  Evaluators: {config['evaluators']}")
+    print(f"  Code evaluators: {['has_output'] + code_evaluators}")
+    if llm_evaluators:
+        print(f"  Pending LLM evaluators (agent): {llm_evaluators}")
 
     try:
         results = client.evaluate(
@@ -192,7 +196,7 @@ def main():
 
         experiment_name = results.experiment_name
 
-        # Calculate mean score
+        # Calculate mean score from code-based evaluators only
         scores = []
         per_example = {}
         for result in results:
@@ -218,10 +222,13 @@ def main():
             "num_examples": len(per_example),
             "num_scores": len(scores),
             "per_example": per_example,
+            "pending_llm_evaluators": llm_evaluators,
         }
 
         print(json.dumps(output))
-        print(f"\nEvaluation complete: {mean_score:.3f} ({len(per_example)} examples)")
+        print(f"\nTarget runs complete: {len(per_example)} examples")
+        if llm_evaluators:
+            print(f"Awaiting evaluator agent for: {llm_evaluators}")
 
     except Exception as e:
         print(f"Evaluation failed: {e}", file=sys.stderr)

package/tools/setup.py

@@ -19,7 +19,7 @@ Usage:
         [--production-project my-prod-project] \
         [--evaluators correctness,conciseness]
 
-Requires: pip install langsmith openevals
+Requires: pip install langsmith
 """
 
 import argparse
@@ -78,16 +78,12 @@ def ensure_langsmith_api_key():
 
 
 def check_dependencies():
-    """Verify langsmith and openevals are installed."""
+    """Verify langsmith is installed."""
     missing = []
     try:
         import langsmith  # noqa: F401
     except ImportError:
         missing.append("langsmith")
-    try:
-        import openevals  # noqa: F401
-    except ImportError:
-        missing.append("openevals")
     return missing
 
 
@@ -177,17 +173,19 @@ def create_empty_dataset(client, dataset_name):
 
 
 def get_evaluators(goals, evaluator_names=None):
-    """Build evaluator list based on optimization goals."""
-    from openevals.llm import create_llm_as_judge
-    from openevals.prompts import CORRECTNESS_PROMPT, CONCISENESS_PROMPT
+    """Build evaluator list based on optimization goals.
 
+    Returns only code-based evaluators. LLM-as-judge evaluators
+    (correctness, conciseness) are handled post-hoc by the
+    evolver-evaluator agent via langsmith-cli.
+    """
     evaluators = []
     evaluator_keys = []
 
-    # Map goals to evaluators
-    goal_map = {
-        "accuracy": ("correctness", CORRECTNESS_PROMPT),
-        "conciseness": ("conciseness", CONCISENESS_PROMPT),
+    # Map goals to evaluator keys (LLM-based are recorded but not instantiated)
+    goal_to_key = {
+        "accuracy": "correctness",
+        "conciseness": "conciseness",
     }
 
     if evaluator_names:
@@ -195,39 +193,33 @@ def get_evaluators(goals, evaluator_names=None):
     else:
         names = []
         for goal in goals:
-            if goal in goal_map:
-                names.append(goal_map[goal][0])
+            if goal in goal_to_key:
+                names.append(goal_to_key[goal])
         if not names:
             names = ["correctness"]  # default
 
+    # Record all evaluator keys (for config) but only instantiate code-based ones
     for name in names:
         if name in ("correctness", "accuracy"):
-            evaluators.append(create_llm_as_judge(
-                prompt=CORRECTNESS_PROMPT,
-                feedback_key="correctness",
-                model="openai:gpt-4.1-mini",
-            ))
             evaluator_keys.append("correctness")
+            # LLM-as-judge — handled by evaluator agent, not here
         elif name in ("conciseness", "brevity"):
-            evaluators.append(create_llm_as_judge(
-                prompt=CONCISENESS_PROMPT,
-                feedback_key="conciseness",
-                model="openai:gpt-4.1-mini",
-            ))
             evaluator_keys.append("conciseness")
+            # LLM-as-judge — handled by evaluator agent, not here
+
+    # Always include has_output
+    def has_output_eval(inputs, outputs, **kwargs):
+        has = bool(outputs and outputs.get("output", outputs.get("answer", "")))
+        return {"key": "has_output", "score": 1.0 if has else 0.0}
+    evaluators.append(has_output_eval)
 
     # Code-based evaluators for latency/tokens
     if "latency" in goals:
-        def latency_eval(inputs, outputs, **kwargs):
-            # Latency is captured in traces, not scored here
-            return {"key": "has_output", "score": 1.0 if outputs else 0.0}
-        evaluators.append(latency_eval)
         evaluator_keys.append("latency")
 
     if "token_efficiency" in goals:
         def token_eval(inputs, outputs, **kwargs):
             output_text = str(outputs.get("output", outputs.get("answer", "")))
-            # Penalize very long outputs (>2000 chars)
             score = min(1.0, 2000 / max(len(output_text), 1))
             return {"key": "token_efficiency", "score": score}
         evaluators.append(token_eval)
@@ -386,18 +378,23 @@ def main():
     print(f"Configuring evaluators for goals: {goals}")
     evaluators, evaluator_keys = get_evaluators(goals, args.evaluators)
     print(f"  Active evaluators: {evaluator_keys}")
+    llm_evaluators = [k for k in evaluator_keys if k in ("correctness", "conciseness")]
+    if llm_evaluators:
+        print(f"  LLM evaluators (agent-based): {llm_evaluators}")
 
-    # Run baseline
+    # Run baseline (code-based evaluators only; LLM scoring done by evaluator agent)
     baseline_experiment = None
     baseline_score = 0.0
     if not args.skip_baseline and count > 0:
-        print(f"Running baseline evaluation ({count} examples)...")
+        print(f"Running baseline target ({count} examples)...")
         try:
             baseline_experiment, baseline_score = run_baseline(
                 client, dataset_name, args.entry_point, evaluators,
             )
-            print(f"  Baseline score: {baseline_score:.3f}")
+            print(f"  Baseline has_output score: {baseline_score:.3f}")
             print(f"  Experiment: {baseline_experiment}")
+            if llm_evaluators:
+                print(f"  Note: LLM scoring pending — evaluator agent will run during /evolver:evolve")
         except Exception as e:
             print(f"  Baseline evaluation failed: {e}", file=sys.stderr)
             print("  Continuing with score 0.0")

package/tools/detect_stack.py

@@ -1,173 +0,0 @@
-#!/usr/bin/env python3
-"""Detect the technology stack of a harness by analyzing Python imports via AST.
-
-Usage:
-    detect_stack.py <file_or_directory> [-o output.json]
-
-Maps imports to known libraries and their Context7 IDs for documentation lookup.
-Stdlib-only. No external dependencies.
-"""
-
-import ast
-import json
-import os
-import sys
-
-KNOWN_LIBRARIES = {
-    "langchain": {
-        "context7_id": "/langchain-ai/langchain",
-        "display": "LangChain",
-        "modules": ["langchain", "langchain_core", "langchain_openai",
-                     "langchain_anthropic", "langchain_community"],
-    },
-    "langgraph": {
-        "context7_id": "/langchain-ai/langgraph",
-        "display": "LangGraph",
-        "modules": ["langgraph"],
-    },
-    "llamaindex": {
-        "context7_id": "/run-llama/llama_index",
-        "display": "LlamaIndex",
-        "modules": ["llama_index"],
-    },
-    "openai": {
-        "context7_id": "/openai/openai-python",
-        "display": "OpenAI Python SDK",
-        "modules": ["openai"],
-    },
-    "anthropic": {
-        "context7_id": "/anthropics/anthropic-sdk-python",
-        "display": "Anthropic Python SDK",
-        "modules": ["anthropic"],
-    },
-    "dspy": {
-        "context7_id": "/stanfordnlp/dspy",
-        "display": "DSPy",
-        "modules": ["dspy"],
-    },
-    "crewai": {
-        "context7_id": "/crewAIInc/crewAI",
-        "display": "CrewAI",
-        "modules": ["crewai"],
-    },
-    "autogen": {
-        "context7_id": "/microsoft/autogen",
-        "display": "AutoGen",
-        "modules": ["autogen"],
-    },
-    "chromadb": {
-        "context7_id": "/chroma-core/chroma",
-        "display": "ChromaDB",
-        "modules": ["chromadb"],
-    },
-    "pinecone": {
-        "context7_id": "/pinecone-io/pinecone-python-client",
-        "display": "Pinecone",
-        "modules": ["pinecone"],
-    },
-    "qdrant": {
-        "context7_id": "/qdrant/qdrant",
-        "display": "Qdrant",
-        "modules": ["qdrant_client"],
-    },
-    "weaviate": {
-        "context7_id": "/weaviate/weaviate",
-        "display": "Weaviate",
-        "modules": ["weaviate"],
-    },
-    "fastapi": {
-        "context7_id": "/fastapi/fastapi",
-        "display": "FastAPI",
-        "modules": ["fastapi"],
-    },
-    "flask": {
-        "context7_id": "/pallets/flask",
-        "display": "Flask",
-        "modules": ["flask"],
-    },
-    "pydantic": {
-        "context7_id": "/pydantic/pydantic",
-        "display": "Pydantic",
-        "modules": ["pydantic"],
-    },
-    "pandas": {
-        "context7_id": "/pandas-dev/pandas",
-        "display": "Pandas",
-        "modules": ["pandas"],
-    },
-    "numpy": {
-        "context7_id": "/numpy/numpy",
-        "display": "NumPy",
-        "modules": ["numpy"],
-    },
-}
-
-
-def detect_from_file(filepath):
-    """Analyze imports of a Python file and return detected stack."""
-    with open(filepath) as f:
-        try:
-            tree = ast.parse(f.read())
-        except SyntaxError:
-            return {}
-
-    imports = set()
-    for node in ast.walk(tree):
-        if isinstance(node, ast.Import):
-            for alias in node.names:
-                imports.add(alias.name.split(".")[0])
-        elif isinstance(node, ast.ImportFrom):
-            if node.module:
-                imports.add(node.module.split(".")[0])
-
-    detected = {}
-    for lib_key, lib_info in KNOWN_LIBRARIES.items():
-        found = imports & set(lib_info["modules"])
-        if found:
-            detected[lib_key] = {
-                "context7_id": lib_info["context7_id"],
-                "display": lib_info["display"],
-                "modules_found": sorted(found),
-            }
-
-    return detected
-
-
-def detect_from_directory(directory):
-    """Analyze all .py files in a directory and consolidate the stack."""
-    all_detected = {}
-    for root, dirs, files in os.walk(directory):
-        for f in files:
-            if f.endswith(".py"):
-                filepath = os.path.join(root, f)
-                file_detected = detect_from_file(filepath)
-                for lib_key, lib_info in file_detected.items():
-                    if lib_key not in all_detected:
-                        all_detected[lib_key] = lib_info
-                    else:
-                        existing = set(all_detected[lib_key]["modules_found"])
-                        existing.update(lib_info["modules_found"])
-                        all_detected[lib_key]["modules_found"] = sorted(existing)
-    return all_detected
-
-
-if __name__ == "__main__":
-    import argparse
-
-    parser = argparse.ArgumentParser(description="Detect stack from Python files")
-    parser.add_argument("path", help="File or directory to analyze")
-    parser.add_argument("--output", "-o", help="Output JSON path")
-    args = parser.parse_args()
-
-    if os.path.isfile(args.path):
-        result = detect_from_file(args.path)
-    else:
-        result = detect_from_directory(args.path)
-
-    output = json.dumps(result, indent=2)
-
-    if args.output:
-        with open(args.output, "w") as f:
-            f.write(output)
-    else:
-        print(output)