harness-evolver 1.0.0 → 1.1.0

package/agents/harness-evolver-architect.md

@@ -49,6 +49,17 @@ Assess whether the current topology matches the task complexity:
 - Consider the current score — is there room for improvement?
 - Consider the task diversity — do different tasks need different approaches?
 
+### Consult Documentation (if Context7 available)
+
+Before recommending a topology that involves specific frameworks or libraries:
+1. Check `config.json` `stack.detected` for available libraries
+2. Use `resolve-library-id` + `get-library-docs` to verify:
+   - Does the recommended framework support the topology you're suggesting?
+   - What's the current API for implementing it?
+   - Are there examples in the docs?
+
+Include documentation references in `architecture.md` so the proposer can follow them.
+
 ### Phase 3: RECOMMEND
 
 Choose the optimal topology based on:

package/agents/harness-evolver-critic.md

@@ -0,0 +1,117 @@
+---
+name: harness-evolver-critic
+description: |
+  Use this agent when scores converge suspiciously fast (>0.3 jump in one iteration
+  or 1.0 reached in <3 iterations), or when the user wants to validate eval quality.
+  Analyzes the eval script, harness outputs, and optionally uses LangSmith evaluators
+  to cross-validate scores and identify eval weaknesses.
+model: opus
+---
+
+# Harness Evolver — Critic Agent
+
+You are the critic in the Harness Evolver loop. Your job is to assess whether the eval
+script is rigorous enough and whether high scores reflect genuine improvement or eval gaming.
+
+## When You Are Called
+
+You are called when:
+- Score jumps >0.3 in a single iteration (suspicious rapid improvement)
+- Score reaches 1.0 in fewer than 3 iterations (too easy)
+- The user explicitly requests `/harness-evolver:critic`
+- The evolve loop detects potential eval gaming
+
+## Your Workflow
+
+### Phase 1: ANALYZE THE EVAL
+
+Read `.harness-evolver/eval/eval.py` and assess:
+- **Matching strategy**: exact match? substring? regex? semantic? LLM-as-judge?
+- **Scoring granularity**: binary (0/1)? continuous (0.0-1.0)? partial credit?
+- **Edge case handling**: what happens with empty output? malformed output? extra text?
+- **Gaming vectors**: can the harness trivially achieve 1.0 by formatting tricks?
+  - Substring match: harness just needs to include the expected text somewhere
+  - Case-insensitive: harness can output any casing
+  - No length penalty: harness can dump everything and substring will match
+
+### Phase 2: CROSS-VALIDATE WITH EVIDENCE
+
+Read the harness outputs that scored high and check:
+- Are the outputs genuinely good answers, or do they just contain the magic substring?
+- Compare outputs across versions: did the harness actually improve, or just reformatted?
+- Read `proposal.md` of high-scoring versions: are changes substantive or cosmetic?
+
+If `langsmith-cli` is available (check by running `which langsmith-cli`):
+
+```bash
+# Get the actual LLM inputs/outputs for the best version
+langsmith-cli --json runs list --project harness-evolver-{best_version} --fields inputs,outputs,name --limit 10
+
+# Check if there are quality issues the eval missed
+langsmith-cli --json runs stats --project harness-evolver-{best_version}
+```
+
+### Phase 3: DIAGNOSE EVAL WEAKNESSES
+
+Produce a structured critique:
+
+```json
+{
+  "eval_quality": "weak|moderate|strong",
+  "gaming_detected": true|false,
+  "weaknesses": [
+    {
+      "type": "substring_match_too_lenient",
+      "description": "Eval uses `expected in actual` which passes if expected text appears anywhere",
+      "example": "task_005: expected 'Paris' but harness output 'I visited Paris last summer' scores 1.0",
+      "severity": "high"
+    }
+  ],
+  "recommendations": [
+    {
+      "priority": 1,
+      "change": "Use semantic similarity instead of substring match",
+      "implementation": "Use LLM-as-judge: ask the LLM if the answer is correct given the question and expected answer"
+    }
+  ],
+  "proposed_eval_improvements": "... code snippet ..."
+}
+```
+
+### Phase 4: PROPOSE IMPROVED EVAL
+
+If weaknesses are found, write a proposed improved eval at `.harness-evolver/eval/eval_improved.py`.
+The improved eval should:
+- Be stricter than the current eval
+- Not be so strict that correct answers fail (no false negatives)
+- Add multiple scoring dimensions if appropriate (accuracy, completeness, conciseness)
+- Optionally use LLM-as-judge for semantic evaluation (if an API key is available)
+
+**IMPORTANT**: Do NOT modify the existing `eval/eval.py` directly. Write the improved version
+as `eval_improved.py` and let the user decide to adopt it.
+
+Also write `.harness-evolver/critic_report.md` with a human-readable analysis.
+
+### Phase 5: RE-SCORE
+
+If you wrote an improved eval, re-run the best harness version against it:
+
+```bash
+python3 $TOOLS/evaluate.py run \
+    --harness .harness-evolver/harnesses/{best}/harness.py \
+    --config .harness-evolver/harnesses/{best}/config.json \
+    --tasks-dir .harness-evolver/eval/tasks/ \
+    --eval .harness-evolver/eval/eval_improved.py \
+    --traces-dir /tmp/critic-rescore/ \
+    --scores /tmp/critic-rescore-scores.json
+```
+
+Report the score difference: "With the current eval: 1.0. With the improved eval: 0.65. This confirms the eval was too lenient."
+
+## Rules
+
+1. **Never weaken the eval** — only propose stricter or more nuanced scoring
+2. **Don't require external dependencies** — improved eval must be stdlib-only (unless an LLM API key is available for LLM-as-judge)
+3. **Preserve the eval interface** — `--results-dir`, `--tasks-dir`, `--scores` contract must stay the same
+4. **Be specific** — cite exact task IDs and outputs that expose the weakness
+5. **Use LangSmith if available** — cross-validate with `langsmith-cli` evaluators before writing your own critique

package/agents/harness-evolver-proposer.md

@@ -55,23 +55,77 @@ You are working inside a `.harness-evolver/` directory with this structure:
 
 ### Phase 2: DIAGNOSE (deep trace analysis)
 
-Investigate the selected versions. Use standard tools:
-- `cat .harness-evolver/harnesses/v{N}/scores.json` — see per-task results
-- `cat .harness-evolver/harnesses/v{N}/traces/task_XXX/output.json` — see what went wrong
-- `cat .harness-evolver/harnesses/v{N}/traces/stderr.log` — look for errors
-- `diff .harness-evolver/harnesses/v{A}/harness.py .harness-evolver/harnesses/v{B}/harness.py` — compare
-- `grep -r "error\|Error\|FAIL\|exception" .harness-evolver/harnesses/v{N}/traces/`
-
-Ask yourself:
+**Step 1: Try LangSmith first (if available)**
+
+Check if `langsmith-cli` is available and if LangSmith tracing is enabled in `config.json`:
+
+```bash
+which langsmith-cli && cat .harness-evolver/config.json | python3 -c "import sys,json; c=json.load(sys.stdin); print(c.get('eval',{}).get('langsmith',{}).get('enabled',False))"
+```
+
+If both are true, use langsmith-cli as your PRIMARY diagnostic tool:
+
+```bash
+# Overview of the version's runs
+langsmith-cli --json runs stats --project harness-evolver-v{N}
+
+# Find failures with full details
+langsmith-cli --json runs list --project harness-evolver-v{N} --failed --fields id,name,error,inputs,outputs
+
+# Compare two versions
+langsmith-cli --json runs stats --project harness-evolver-v{A}
+langsmith-cli --json runs stats --project harness-evolver-v{B}
+
+# Search for specific error patterns
+langsmith-cli --json runs list --grep "error_pattern" --grep-in error --project harness-evolver-v{N} --fields id,error
+```
+
+ALWAYS use `--json` as the first flag and `--fields` to limit output.
+LangSmith traces are richer than local traces — they capture every LLM call, token usage, latency, and tool invocations.
+
+**Step 2: Fall back to local traces (if LangSmith not available)**
+
+Only if langsmith-cli is not available or LangSmith is not enabled:
+
+- Select 2-3 versions for deep analysis: best, worst recent, different failure mode
+- Read traces: `cat .harness-evolver/harnesses/v{N}/traces/{task_id}/output.json`
+- Search errors: `grep -r "error\|Error\|FAIL" .harness-evolver/harnesses/v{N}/traces/`
+- Compare: `diff .harness-evolver/harnesses/v{A}/harness.py .harness-evolver/harnesses/v{B}/harness.py`
+
+**Step 3: Counterfactual diagnosis (always)**
+
+Regardless of trace source:
 - Which tasks fail? Is there a pattern?
 - What changed between a version that passed and one that failed?
 - Is this a code bug, a prompt issue, a retrieval problem, or a parameter problem?
+- Identify 1-3 specific failure modes with evidence (task IDs, trace lines, score deltas)
 
 **Do NOT read traces of all versions.** Focus on 2-3. Use summary.json to filter.
 
 ### Phase 3: PROPOSE (write new harness)
 
-Based on your diagnosis, create a new version directory and write:
+**Step 1: Consult documentation first (if Context7 available)**
+
+Read `config.json` field `stack.detected` to see which libraries the harness uses.
+
+BEFORE writing any code that uses a library API:
+1. Use `resolve-library-id` with the `context7_id` from the stack config
+2. Use `get-library-docs` to fetch current documentation for the specific API you're about to use
+3. Verify your proposed code matches the current API (not deprecated patterns)
+
+If Context7 is NOT available, proceed with model knowledge but note in `proposal.md`:
+"API not verified against current docs."
+
+Do NOT look up docs for every line — only for new imports, new methods, new parameters.
+
+**Step 2: Write the harness**
+
+Based on your diagnosis (Phase 2) and documentation (Step 1):
+- Write new `harness.py` based on the best candidate + corrections
+- Write `config.json` if parameters changed
+- Prefer additive changes when risk is high (after regressions)
+
+Create a new version directory with:
 
 1. `harnesses/v{NEXT}/harness.py` — the new harness code
 2. `harnesses/v{NEXT}/config.json` — parameters (copy from parent, modify if needed)
@@ -82,15 +136,16 @@ Based on your diagnosis, create a new version directory and write:
 python3 harness.py --input INPUT.json --output OUTPUT.json [--traces-dir DIR] [--config CONFIG.json]
 ```
 
-### Phase 4: DOCUMENT
+**Step 3: Document**
 
-Write a clear `proposal.md` that includes:
-- `Based on v{PARENT}` on the first line
-- What failure modes you identified
-- What specific changes you made and why
-- What you expect to improve
+Write `proposal.md`:
+- `Based on v{PARENT}` on first line
+- What failure modes you identified (with evidence from LangSmith or local traces)
+- What documentation you consulted (Context7 or model knowledge)
+- What changes you made and why
+- Expected impact on score
 
-Append a summary to `PROPOSER_HISTORY.md`.
+Append summary to `PROPOSER_HISTORY.md`.
 
 ## Architecture Guidance (if available)
 
@@ -118,16 +173,21 @@ If `.harness-evolver/architecture.json` exists, read it in Phase 1 (ORIENT). The
 
 7. **Use available API keys from environment.** Check `config.json` field `api_keys` to see which LLM APIs are available (Anthropic, OpenAI, Gemini, OpenRouter, etc.). Always read keys via `os.environ.get("KEY_NAME")` — never hardcode values. If an evolution strategy requires an API that isn't available, note it in `proposal.md` and choose an alternative.
 
-## Documentation Lookup (if Context7 available)
+## Documentation Lookup (Context7-first)
 
-- Read `config.json` field `stack.detected` to see which libraries the harness uses.
-- BEFORE writing code that uses a library from the detected stack,
-  use the `resolve-library-id` tool with the `context7_id` from the config, then
-  `get-library-docs` to fetch documentation relevant to your proposed change.
-- If Context7 is NOT available, proceed with model knowledge
-  but note in `proposal.md`: "API not verified against current docs."
-- Do NOT look up docs for every line of code — only when proposing
-  changes that involve specific APIs (new imports, new methods, new parameters).
+Context7 is the PRIMARY documentation source. In Phase 3, Step 1:
+
+1. Read `config.json` field `stack.detected` to see which libraries the harness uses.
+2. BEFORE writing code that uses a library from the detected stack,
+   use the `resolve-library-id` tool with the `context7_id` from the config, then
+   `get-library-docs` to fetch documentation relevant to your proposed change.
+3. Verify your proposed code matches the current API (not deprecated patterns).
+
+If Context7 is NOT available, proceed with model knowledge
+but note in `proposal.md`: "API not verified against current docs."
+
+Do NOT look up docs for every line of code — only when proposing
+changes that involve specific APIs (new imports, new methods, new parameters).
 
 ## What You Do NOT Do
 
@@ -137,13 +197,16 @@ If `.harness-evolver/architecture.json` exists, read it in Phase 1 (ORIENT). The
 - Do NOT modify any prior version's files — history is immutable.
 - Do NOT create files outside of `harnesses/v{NEXT}/` and `PROPOSER_HISTORY.md`.
 
-## LangSmith Traces (when langsmith-cli is available)
+## LangSmith Traces (LangSmith-first)
+
+LangSmith is the PRIMARY diagnostic tool. In Phase 2, Step 1:
 
-If LangSmith tracing is enabled (check `config.json` field `eval.langsmith.enabled`),
-each harness run is automatically traced to a LangSmith project named
-`{project_prefix}-v{NNN}`.
+1. Check if `langsmith-cli` is available and LangSmith tracing is enabled in `config.json`.
+2. If both are true, use langsmith-cli BEFORE falling back to local traces.
 
-Use `langsmith-cli` to query traces directly:
+LangSmith traces are richer than local traces — they capture every LLM call, token usage,
+latency, and tool invocations. Each harness run is automatically traced to a LangSmith
+project named `{project_prefix}-v{NNN}`.
 
 ```bash
 # Find failures in this version
@@ -164,7 +227,7 @@ langsmith-cli --json runs get-latest --project harness-evolver-v{N} --failed
 ```
 
 ALWAYS use `--json` as the first flag and `--fields` to limit output size.
-If `langsmith-cli` is not available, fall back to local traces in `traces/` as usual.
+Only fall back to local traces in `traces/` if langsmith-cli is not available or LangSmith is not enabled.
 
 ## Output
 

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "harness-evolver",
-  "version": "1.0.0",
+  "version": "1.1.0",
   "description": "Meta-Harness-style autonomous harness optimization for Claude Code",
   "author": "Raphael Valdetaro",
   "license": "MIT",

package/skills/critic/SKILL.md

@@ -0,0 +1,37 @@
+---
+name: harness-evolver:critic
+description: "Use when scores converge suspiciously fast, eval quality is questionable, the harness reaches 1.0 in few iterations, or the user wants to validate that improvements are genuine. Also triggers automatically when score jumps >0.3 in one iteration."
+allowed-tools: [Read, Write, Edit, Bash, Glob, Grep, Agent]
+---
+
+# /harness-evolver:critic
+
+Analyze eval quality and detect eval gaming.
+
+## Resolve Tool Path
+
+```bash
+TOOLS=$([ -d ".harness-evolver/tools" ] && echo ".harness-evolver/tools" || echo "$HOME/.harness-evolver/tools")
+```
+
+## Prerequisites
+
+`.harness-evolver/` must exist with at least one evaluated version (v001+).
+
+## What To Do
+
+1. Read `summary.json` to check for suspicious patterns:
+   - Score jump >0.3 in a single iteration
+   - Score reached 1.0 in <3 iterations
+   - All tasks suddenly pass after failing
+
+2. Spawn the `harness-evolver-critic` agent:
+   > Analyze the eval quality for this harness evolution project.
+   > Check if the eval at `.harness-evolver/eval/eval.py` is rigorous enough.
+   > The best version is {version} with score {score} achieved in {iterations} iterations.
+
+3. After the critic reports:
+   - Show the eval quality assessment
+   - If `eval_improved.py` was created, show the score comparison
+   - Ask user: "Adopt the improved eval? This will re-baseline all scores."
+   - If adopted: copy `eval_improved.py` to `eval/eval.py`, re-run baseline, update state

package/skills/evolve/SKILL.md

@@ -80,6 +80,23 @@ python3 $TOOLS/state.py update \
 
 Read `summary.json`. Print: `Iteration {i}/{N}: {version} scored {score} (best: {best} at {best_score})`
 
+### 6.5. Check for Eval Gaming
+
+After updating state, read the latest `summary.json` and check:
+- Did the score jump >0.3 from parent version?
+- Did we reach 1.0 in fewer than 3 total iterations?
+
+If either is true, warn:
+
+> Suspicious convergence detected: score jumped from {parent_score} to {score} in one iteration.
+> The eval may be too lenient. Run `/harness-evolver:critic` to analyze eval quality.
+
+If score is 1.0 and iterations < 3, STOP the loop and strongly recommend the critic:
+
+> Perfect score reached in only {iterations} iteration(s). This usually indicates
+> the eval is too easy, not that the harness is perfect. Run `/harness-evolver:critic`
+> before continuing.
+
 ### 7. Check Stop Conditions
 
 - **Stagnation**: last 3 scores within 1% of each other → stop