harness-evolve 0.1.0__tar.gz

harness_evolve-0.1.0/.github/workflows/release.yml

@@ -0,0 +1,40 @@
+name: release
+
+# Publishes to PyPI when a version tag (v0.1.0, ...) is pushed. Uses PyPI trusted publishing:
+# no token is stored anywhere; PyPI trusts this workflow in this repo via the "pypi" environment.
+
+on:
+  push:
+    tags: ["v*"]
+
+jobs:
+  build:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-python@v5
+        with:
+          python-version: "3.13"
+      - name: Tag must match the package version
+        run: |
+          python -c "import tomllib, os; v = tomllib.load(open('pyproject.toml', 'rb'))['project']['version']; t = os.environ['GITHUB_REF_NAME'].removeprefix('v'); assert v == t, f'tag {t} != pyproject version {v}'"
+      - run: pip install build twine
+      - run: python -m build
+      - run: twine check dist/*
+      - uses: actions/upload-artifact@v4
+        with:
+          name: dist
+          path: dist/
+
+  publish:
+    needs: build
+    runs-on: ubuntu-latest
+    environment: pypi
+    permissions:
+      id-token: write
+    steps:
+      - uses: actions/download-artifact@v4
+        with:
+          name: dist
+          path: dist/
+      - uses: pypa/gh-action-pypi-publish@release/v1

harness_evolve-0.1.0/.github/workflows/tests.yml

@@ -0,0 +1,30 @@
+name: tests
+
+on:
+  push:
+    branches: [main]
+  pull_request:
+
+jobs:
+  test:
+    runs-on: ubuntu-latest
+    strategy:
+      matrix:
+        python-version: ["3.11", "3.12", "3.13"]
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-python@v5
+        with:
+          python-version: ${{ matrix.python-version }}
+      - run: pip install -e ".[dev]"
+      - run: pytest -q
+
+  lint:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-python@v5
+        with:
+          python-version: "3.13"
+      - run: pip install ruff
+      - run: ruff check .

harness_evolve-0.1.0/.gitignore

@@ -0,0 +1,8 @@
+.venv/
+__pycache__/
+*.pyc
+.pytest_cache/
+runs/
+dist/
+*.egg-info/
+examples/math_agent/data/

harness_evolve-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Siddharth Sukumar
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

harness_evolve-0.1.0/PKG-INFO

@@ -0,0 +1,207 @@
+Metadata-Version: 2.5
+Name: harness-evolve
+Version: 0.1.0
+Summary: Evolve an agent's harness (prompts, tool descriptions, settings) from its own failure traces, accepting only changes that beat measured run-to-run noise.
+Author: Siddharth Sukumar
+License-Expression: MIT
+License-File: LICENSE
+Keywords: agents,evals,harness-engineering,llm,prompt-optimization
+Requires-Python: >=3.11
+Provides-Extra: dev
+Requires-Dist: litellm>=1.40; extra == 'dev'
+Requires-Dist: pytest>=8; extra == 'dev'
+Requires-Dist: ruff>=0.15; extra == 'dev'
+Provides-Extra: litellm
+Requires-Dist: litellm>=1.40; extra == 'litellm'
+Description-Content-Type: text/markdown
+
+# harness-evolve
+
+[![tests](https://github.com/sidreddy88/harness-evolve/actions/workflows/tests.yml/badge.svg)](https://github.com/sidreddy88/harness-evolve/actions/workflows/tests.yml)
+
+Improve an AI agent's harness (its prompts, tool descriptions and settings) from the agent's own failure
+traces, and keep a change only when it beats measured run-to-run noise.
+
+Agents are noisy: run the same agent on the same task twice and the result can flip. Tune a prompt,
+rerun, see +3 points, and you may have measured nothing. harness-evolve measures that noise first,
+then accepts an edit only if it clears it, checks it doesn't break the cases that already work, and
+reports the result on cases it never saw, against the cheapest alternative: just running the original
+agent twice.
+
+## Results
+
+Used on a production code-diagnosis agent ([agent-platform](https://github.com/sidreddy88/agent-platform)), on SWE-bench Verified with
+DeepSeek-V4.1-Flash:
+
+![Held-out accuracy and cost per task: original one run 78.8% at $0.033, original best of 2 runs 89.9% at $0.066, evolved one run 89.9% at $0.040](https://raw.githubusercontent.com/sidreddy88/harness-evolve/main/docs/results.png)
+
+| Held-out: 66 cases from repositories the optimizer never saw | Original harness | Evolved harness |
+|---|---|---|
+| Finds the file the fix must change (single run) | 78.8% | **89.9%** (+11.1, 95% CI +6.1 to +16.2) |
+| Gives up with no answer | 19.7% | 5.6% |
+| Solves it on all 3 trials (reliability) | 60.6% | 78.8% |
+
+The evolved harness matched running the original twice (best-of-2) at about 60% of the cost.
+Two edits were accepted in a 7.4-hour run, and the evolved harness went to production.
+
+It also says no. An earlier run of the same optimizer on Claude Sonnet 5 accepted nothing, and so
+did a three-round run on the same project's fix agent: every candidate edit either stayed inside the
+noise band or broke a guard case. Halfway through that run, one edit led 7 to 4, and a single
+measurement would have reported it as a gain.
+
+## Quickstart
+
+The example is a small math agent (a calculator tool and a submit tool) on GSM-Hard, GSM8K questions
+with large, awkward numbers that leave room to improve.
+
+```bash
+git clone https://github.com/sidreddy88/harness-evolve && cd harness-evolve
+pip install -e ".[litellm]"
+python examples/math_agent/fetch_data.py      # downloads 80 problems
+export TOGETHER_API_KEY=...                   # or any litellm provider; set MODEL to match
+harness-evolve run examples/math_agent/config.py --run-dir runs/math-1 --budget 2
+harness-evolve status runs/math-1
+harness-evolve diff runs/math-1               # accepted edits, as diffs to review
+```
+
+What to expect (our run, DeepSeek-V4.1-Flash, $1.06): the starting harness solved 76% of the 50 evolve
+problems, with a measured noise band of ±5.6 points. The proposer read the trajectories and aimed three
+edits at real failures (runs that never submit an answer, then wrong arithmetic); each gained +2 to +3
+points, inside the noise, so none was accepted. With 50 cases only bigger gains are detectable; more
+cases narrow the band.
+
+`--budget` is a hard cap in dollars. A run that stops (budget, Ctrl-C, a provider outage) resumes with
+the same command, and nothing already paid for is paid again.
+
+## How a run works
+
+```
+round 0   run the starting harness on the evolve cases, 2 trials each
+          -> the noise band (how much the pass rate moves between identical runs)
+round r   propose one edit, aimed at failures in the trajectories (GEPA-style)
+          -> structural checks (free)
+          -> a critic screens for leakage, gaming and weakened safeguards (cheap)
+          -> evaluate the candidate (the only expensive step)
+          -> accept only if it beats the noise band, or is cheaper beyond cost noise
+             at the same pass rate, and no guard case regresses
+final     original vs evolved on held-out cases, k trials each, including the
+          original's best-of-2 (the matched-budget rerun baseline)
+```
+
+Accepted edits are written as diffs under `runs/<id>/proposals/` for you to review; your code is never
+changed automatically.
+
+Built for long unattended runs: every phase is checkpointed; every case's result is cached by the
+candidate's content hash; a hard budget is checked before each case; provider failures (billing, auth,
+rate limits) stop the run instead of being scored as failures; tripwires pause it when measurements
+look broken; a watchdog exits cleanly if it stops making progress.
+
+## Using it on your agent
+
+You need five things.
+
+**1. A harness directory your agent reads at run time**
+
+```
+my_harness/
+  system.prompt              # templates, rendered with str.format
+  task.prompt
+  tool_descriptions.json     # {"search": "Search the docs ..."}
+  settings.json              # {"max_turns": 12, "_doc": {"max_turns": "turn budget"}}
+```
+
+Load it with `harness_evolve.load_harness(path)`. The optimizer edits copies of these files, never
+adds or deletes files, and checks every edit still loads.
+
+**2. An evaluator**
+
+```python
+class MyEvaluator:
+    async def evaluate(self, harness_dir, case_ids, trials) -> EvalOutcome:
+        ...  # run your agent with harness_dir on each case, `trials` times
+```
+
+Return pass/fail and cost per trial, and trajectories if you can: the proposer reads them to find
+out why runs fail. Raise `ProviderFailure` for billing, auth or outage errors. Never let the right
+answer reach the proposer: put it after `evidence.HIDDEN_MARKER` in a trajectory's `detail`.
+See `examples/math_agent/evaluator.py`.
+
+**3. A profile**
+
+```python
+from harness_evolve import AgentProfile, profiles
+profiles.use(AgentProfile.build(
+    name="support-bot", harness_root="my_harness/",
+    agent_description="Answers support tickets with search and read tools; finishes by calling reply.",
+    safety_mechanisms=["citing a source document for every claim"],
+    setting_bounds={"max_turns": (3, 20)}))
+```
+
+The description and safety mechanisms become the proposer's and critic's instructions; setting
+bounds keep an edit from "saving cost" by starving the agent.
+
+**4. Cases**: an evolve set (what it improves on, ideally 50+), optional guard cases (must keep
+passing) and held-out cases (never seen during evolution, for the final report).
+
+**5. A model for the proposer and critic, and a budget**: `litellm_llm("model-id", prices=(in, out))`
+or any async `(system, prompt) -> LLMReply(text, cost_usd)`.
+
+Then:
+
+```python
+opt = Optimizer("runs/r1", "my_harness/", OptimizerConfig(evolve_cases=..., budget_usd=50),
+                evaluator=MyEvaluator(), proposer_llm=llm, critic_llm=llm, critic_patterns=[])
+asyncio.run(opt.run_until_stopped())
+```
+
+Or put a `build(run_dir, budget_usd, rounds)` function in a config file and use the CLI, as the
+example does.
+
+## Benchmarking the optimizer
+
+`harness-evolve bench` measures the optimizer itself, so a change to its method is judged by
+evidence, not opinion. Known-answer scenarios have a right outcome: a scripted proposer puts one known
+edit in front of the acceptance rules. A real fix to a planted flaw must be accepted, a harmful edit
+must be rejected, and a neutral rewording must not be accepted (a false accept). Open scenarios run the
+real proposer and report what it finds and what it gains on held-out cases, per dollar.
+
+```bash
+python examples/math_agent/fetch_data.py --n 260 --out examples/math_agent/data/problems-bench.jsonl
+harness-evolve bench examples/math_agent/bench.py --out runs/bench-1
+```
+
+Results go to `runs/bench-1/bench.md`. Our first run (DeepSeek-V4.1-Flash, $9.37 in total):
+
+| Scenario | Expected | Result |
+|---|---|---|
+| accept-good: planted flaw (`max_turns` 1) + scripted fix | accepted | PASS: +49 points |
+| reject-bad: scripted "round answers to the nearest hundred" | not accepted | PASS: −21.5 points, rejected |
+| reject-noise: scripted neutral rewording | not accepted | PASS: rejected |
+| find-flaw: planted flaw, real proposer | (reported) | found the flaw from trajectories: held-out 28.3% → 79.0% |
+| standard: normal harness, 200 problems, real proposer | (reported) | one prompt edit accepted: held-out accuracy unchanged (83.3% → 83.9%), ~11% cheaper |
+
+The first version of reject-bad ("skip the calculator and estimate") was accepted: it didn't hurt this
+model at all and was 13% cheaper, so the optimizer was right and the test was wrong. A known-answer
+scenario's premise has to be measured, not assumed. Write your own scenarios with `harness_evolve.bench.Scenario`
+and `scripted_llm` (see `examples/math_agent/bench.py`).
+
+## Limits
+
+- It needs an automatic pass/fail check per case. No grader, no optimizer.
+- A noisy agent needs more cases or trials before any gain is detectable; the noise band will tell
+  you, and "nothing accepted" is a real answer.
+- Every candidate costs a full evaluation. Slow or expensive agents make slow, expensive runs.
+
+## Credits
+
+Full mapping of each component to its source, and what we changed: [docs/methods.md](docs/methods.md).
+
+
+- Reflective, trace-driven proposals: GEPA (Agrawal et al., "GEPA: Reflective Prompt Evolution Can
+  Outperform Reinforcement Learning", ICLR 2026).
+- The critic and the one-edit-per-round discipline: RRSI (google-research/rrsi, Apache-2.0); the
+  critic's reject classes are adapted from its prompt.
+- Comparing against reruns: AI2's finding that harness evolution rarely beats simply rerunning
+  (arXiv 2607.12227), which is why the final report includes the matched-budget rerun baseline.
+
+MIT license.

harness_evolve-0.1.0/README.md

@@ -0,0 +1,190 @@
+# harness-evolve
+
+[![tests](https://github.com/sidreddy88/harness-evolve/actions/workflows/tests.yml/badge.svg)](https://github.com/sidreddy88/harness-evolve/actions/workflows/tests.yml)
+
+Improve an AI agent's harness (its prompts, tool descriptions and settings) from the agent's own failure
+traces, and keep a change only when it beats measured run-to-run noise.
+
+Agents are noisy: run the same agent on the same task twice and the result can flip. Tune a prompt,
+rerun, see +3 points, and you may have measured nothing. harness-evolve measures that noise first,
+then accepts an edit only if it clears it, checks it doesn't break the cases that already work, and
+reports the result on cases it never saw, against the cheapest alternative: just running the original
+agent twice.
+
+## Results
+
+Used on a production code-diagnosis agent ([agent-platform](https://github.com/sidreddy88/agent-platform)), on SWE-bench Verified with
+DeepSeek-V4.1-Flash:
+
+![Held-out accuracy and cost per task: original one run 78.8% at $0.033, original best of 2 runs 89.9% at $0.066, evolved one run 89.9% at $0.040](https://raw.githubusercontent.com/sidreddy88/harness-evolve/main/docs/results.png)
+
+| Held-out: 66 cases from repositories the optimizer never saw | Original harness | Evolved harness |
+|---|---|---|
+| Finds the file the fix must change (single run) | 78.8% | **89.9%** (+11.1, 95% CI +6.1 to +16.2) |
+| Gives up with no answer | 19.7% | 5.6% |
+| Solves it on all 3 trials (reliability) | 60.6% | 78.8% |
+
+The evolved harness matched running the original twice (best-of-2) at about 60% of the cost.
+Two edits were accepted in a 7.4-hour run, and the evolved harness went to production.
+
+It also says no. An earlier run of the same optimizer on Claude Sonnet 5 accepted nothing, and so
+did a three-round run on the same project's fix agent: every candidate edit either stayed inside the
+noise band or broke a guard case. Halfway through that run, one edit led 7 to 4, and a single
+measurement would have reported it as a gain.
+
+## Quickstart
+
+The example is a small math agent (a calculator tool and a submit tool) on GSM-Hard, GSM8K questions
+with large, awkward numbers that leave room to improve.
+
+```bash
+git clone https://github.com/sidreddy88/harness-evolve && cd harness-evolve
+pip install -e ".[litellm]"
+python examples/math_agent/fetch_data.py      # downloads 80 problems
+export TOGETHER_API_KEY=...                   # or any litellm provider; set MODEL to match
+harness-evolve run examples/math_agent/config.py --run-dir runs/math-1 --budget 2
+harness-evolve status runs/math-1
+harness-evolve diff runs/math-1               # accepted edits, as diffs to review
+```
+
+What to expect (our run, DeepSeek-V4.1-Flash, $1.06): the starting harness solved 76% of the 50 evolve
+problems, with a measured noise band of ±5.6 points. The proposer read the trajectories and aimed three
+edits at real failures (runs that never submit an answer, then wrong arithmetic); each gained +2 to +3
+points, inside the noise, so none was accepted. With 50 cases only bigger gains are detectable; more
+cases narrow the band.
+
+`--budget` is a hard cap in dollars. A run that stops (budget, Ctrl-C, a provider outage) resumes with
+the same command, and nothing already paid for is paid again.
+
+## How a run works
+
+```
+round 0   run the starting harness on the evolve cases, 2 trials each
+          -> the noise band (how much the pass rate moves between identical runs)
+round r   propose one edit, aimed at failures in the trajectories (GEPA-style)
+          -> structural checks (free)
+          -> a critic screens for leakage, gaming and weakened safeguards (cheap)
+          -> evaluate the candidate (the only expensive step)
+          -> accept only if it beats the noise band, or is cheaper beyond cost noise
+             at the same pass rate, and no guard case regresses
+final     original vs evolved on held-out cases, k trials each, including the
+          original's best-of-2 (the matched-budget rerun baseline)
+```
+
+Accepted edits are written as diffs under `runs/<id>/proposals/` for you to review; your code is never
+changed automatically.
+
+Built for long unattended runs: every phase is checkpointed; every case's result is cached by the
+candidate's content hash; a hard budget is checked before each case; provider failures (billing, auth,
+rate limits) stop the run instead of being scored as failures; tripwires pause it when measurements
+look broken; a watchdog exits cleanly if it stops making progress.
+
+## Using it on your agent
+
+You need five things.
+
+**1. A harness directory your agent reads at run time**
+
+```
+my_harness/
+  system.prompt              # templates, rendered with str.format
+  task.prompt
+  tool_descriptions.json     # {"search": "Search the docs ..."}
+  settings.json              # {"max_turns": 12, "_doc": {"max_turns": "turn budget"}}
+```
+
+Load it with `harness_evolve.load_harness(path)`. The optimizer edits copies of these files, never
+adds or deletes files, and checks every edit still loads.
+
+**2. An evaluator**
+
+```python
+class MyEvaluator:
+    async def evaluate(self, harness_dir, case_ids, trials) -> EvalOutcome:
+        ...  # run your agent with harness_dir on each case, `trials` times
+```
+
+Return pass/fail and cost per trial, and trajectories if you can: the proposer reads them to find
+out why runs fail. Raise `ProviderFailure` for billing, auth or outage errors. Never let the right
+answer reach the proposer: put it after `evidence.HIDDEN_MARKER` in a trajectory's `detail`.
+See `examples/math_agent/evaluator.py`.
+
+**3. A profile**
+
+```python
+from harness_evolve import AgentProfile, profiles
+profiles.use(AgentProfile.build(
+    name="support-bot", harness_root="my_harness/",
+    agent_description="Answers support tickets with search and read tools; finishes by calling reply.",
+    safety_mechanisms=["citing a source document for every claim"],
+    setting_bounds={"max_turns": (3, 20)}))
+```
+
+The description and safety mechanisms become the proposer's and critic's instructions; setting
+bounds keep an edit from "saving cost" by starving the agent.
+
+**4. Cases**: an evolve set (what it improves on, ideally 50+), optional guard cases (must keep
+passing) and held-out cases (never seen during evolution, for the final report).
+
+**5. A model for the proposer and critic, and a budget**: `litellm_llm("model-id", prices=(in, out))`
+or any async `(system, prompt) -> LLMReply(text, cost_usd)`.
+
+Then:
+
+```python
+opt = Optimizer("runs/r1", "my_harness/", OptimizerConfig(evolve_cases=..., budget_usd=50),
+                evaluator=MyEvaluator(), proposer_llm=llm, critic_llm=llm, critic_patterns=[])
+asyncio.run(opt.run_until_stopped())
+```
+
+Or put a `build(run_dir, budget_usd, rounds)` function in a config file and use the CLI, as the
+example does.
+
+## Benchmarking the optimizer
+
+`harness-evolve bench` measures the optimizer itself, so a change to its method is judged by
+evidence, not opinion. Known-answer scenarios have a right outcome: a scripted proposer puts one known
+edit in front of the acceptance rules. A real fix to a planted flaw must be accepted, a harmful edit
+must be rejected, and a neutral rewording must not be accepted (a false accept). Open scenarios run the
+real proposer and report what it finds and what it gains on held-out cases, per dollar.
+
+```bash
+python examples/math_agent/fetch_data.py --n 260 --out examples/math_agent/data/problems-bench.jsonl
+harness-evolve bench examples/math_agent/bench.py --out runs/bench-1
+```
+
+Results go to `runs/bench-1/bench.md`. Our first run (DeepSeek-V4.1-Flash, $9.37 in total):
+
+| Scenario | Expected | Result |
+|---|---|---|
+| accept-good: planted flaw (`max_turns` 1) + scripted fix | accepted | PASS: +49 points |
+| reject-bad: scripted "round answers to the nearest hundred" | not accepted | PASS: −21.5 points, rejected |
+| reject-noise: scripted neutral rewording | not accepted | PASS: rejected |
+| find-flaw: planted flaw, real proposer | (reported) | found the flaw from trajectories: held-out 28.3% → 79.0% |
+| standard: normal harness, 200 problems, real proposer | (reported) | one prompt edit accepted: held-out accuracy unchanged (83.3% → 83.9%), ~11% cheaper |
+
+The first version of reject-bad ("skip the calculator and estimate") was accepted: it didn't hurt this
+model at all and was 13% cheaper, so the optimizer was right and the test was wrong. A known-answer
+scenario's premise has to be measured, not assumed. Write your own scenarios with `harness_evolve.bench.Scenario`
+and `scripted_llm` (see `examples/math_agent/bench.py`).
+
+## Limits
+
+- It needs an automatic pass/fail check per case. No grader, no optimizer.
+- A noisy agent needs more cases or trials before any gain is detectable; the noise band will tell
+  you, and "nothing accepted" is a real answer.
+- Every candidate costs a full evaluation. Slow or expensive agents make slow, expensive runs.
+
+## Credits
+
+Full mapping of each component to its source, and what we changed: [docs/methods.md](docs/methods.md).
+
+
+- Reflective, trace-driven proposals: GEPA (Agrawal et al., "GEPA: Reflective Prompt Evolution Can
+  Outperform Reinforcement Learning", ICLR 2026).
+- The critic and the one-edit-per-round discipline: RRSI (google-research/rrsi, Apache-2.0); the
+  critic's reject classes are adapted from its prompt.
+- Comparing against reruns: AI2's finding that harness evolution rarely beats simply rerunning
+  (arXiv 2607.12227), which is why the final report includes the matched-budget rerun baseline.
+
+MIT license.

harness_evolve-0.1.0/docs/make_chart.py

@@ -0,0 +1,46 @@
+"""Draws docs/results.png from the held-out numbers in the README (agent-platform's
+diagnosis agent, SWE-bench Verified, 66 cases from repositories the optimizer never saw,
+DeepSeek-V4.1-Flash). Run: python docs/make_chart.py"""
+from pathlib import Path
+
+import matplotlib
+
+matplotlib.use("Agg")
+import matplotlib.pyplot as plt
+import matplotlib.ticker
+
+LABELS = ["Original\n1 run", "Original\nbest of 2 runs*", "Evolved\n1 run"]
+ACC = [78.8, 89.9, 89.9]
+COST = [0.033, 0.066, 0.040]
+COLORS = ["#9aa5b1", "#9aa5b1", "#2563eb"]
+
+fig, (a1, a2) = plt.subplots(1, 2, figsize=(9, 3.6), dpi=200)
+for ax in (a1, a2):
+    ax.spines[["top", "right"]].set_visible(False)
+    ax.tick_params(axis="x", length=0)
+
+b = a1.bar(LABELS, ACC, color=COLORS, width=0.6)
+a1.set_ylim(0, 100)
+a1.yaxis.set_major_formatter(matplotlib.ticker.PercentFormatter(decimals=0))
+a1.set_title("Held-out accuracy", fontsize=11, loc="left")
+for rect, v in zip(b, ACC, strict=True):
+    a1.text(rect.get_x() + rect.get_width() / 2, v / 2, f"{v:.1f}%", ha="center", fontsize=10,
+            color="white", fontweight="bold")
+a1.errorbar(2, 89.9, yerr=[[89.9 - (78.8 + 6.1)], [(78.8 + 16.2) - 89.9]], fmt="none",
+            ecolor="#1e3a8a", capsize=4, lw=1)
+
+b = a2.bar(LABELS, COST, color=COLORS, width=0.6)
+a2.set_ylim(0, 0.08)
+a2.yaxis.set_major_formatter(matplotlib.ticker.FuncFormatter(lambda v, _: f"${v:.2f}"))
+a2.set_title("Cost per task", fontsize=11, loc="left")
+for rect, v in zip(b, COST, strict=True):
+    a2.text(rect.get_x() + rect.get_width() / 2, v / 2, f"${v:.3f}", ha="center", fontsize=10,
+            color="white", fontweight="bold")
+
+fig.text(0.01, 0.01, "66 SWE-bench Verified cases from repositories the optimizer never saw; DeepSeek-V4.1-Flash. "
+         "Error bar: 95% CI of the gain over one run.\n*Best of 2 assumes a perfect picker between the two answers.",
+         fontsize=7.5, color="#52606d")
+fig.tight_layout(rect=(0, 0.08, 1, 1))
+out = Path(__file__).parent / "results.png"
+fig.savefig(out)
+print(out)

harness_evolve-0.1.0/docs/methods.md

@@ -0,0 +1,71 @@
+# Methods: what harness-evolve does, and where each idea comes from
+
+Each part of the optimizer, the file it lives in, the source it comes from, and what we changed. Settings
+are fields of `OptimizerConfig` (`loop.py`) or `AcceptanceConfig` (`acceptance.py`); "off by default" means
+you turn it on there.
+
+## The loop (`loop.py`)
+
+Round 0 runs the starting harness on the evolve cases (2 trials each) and calibrates the noise band. Each
+round then proposes one edit, checks it for free, screens it with a critic, evaluates it (the only
+expensive step), and decides with fixed rules. An optional final phase compares the original and evolved
+harness on held-out cases. Every phase is checkpointed; every case result is cached by the candidate's
+content hash, so a stopped run resumes without paying twice.
+
+## Proposing an edit (`proposer.py`, `evidence.py`)
+
+| What | How | Source |
+|---|---|---|
+| Reflective, trace-driven proposals | The proposer reads the agent's trajectories (tool calls, outputs, verdicts, cost) and the edit history, diagnoses what went wrong in words, and targets an edit at that failure, instead of mutating a score blindly | GEPA: Agrawal et al., "GEPA: Reflective Prompt Evolution Can Outperform Reinforcement Learning", ICLR 2026. Ours is GEPA-style (hand-built; no GEPA library, no candidate pool) |
+| One mechanism per candidate | Each candidate is one component, one hypothesis, one mechanism, so every measured change traces to one idea | RRSI's annealed edit budget at its floor (b = 1): Xia et al., "RRSI: Regularized Recursive Self-Improvement of Agent Harnesses", arXiv 2609.24972 |
+| Two candidates per round | The proposer returns two alternatives with different mechanisms; if the critic rejects the preferred one, the other is tried | RRSI (two proposals per round) |
+| Search/replace edits | Edits are exact find/replace operations on harness files, not full rewrites, so a long prompt isn't reproduced (and silently drifted) to change one paragraph | Ours |
+| Settings shown as behaviour | Each setting appears with what it does and its allowed range | Ours: in our first runs the proposer never touched settings until they were shown this way |
+| Answers kept from the proposer | Anything after `HIDDEN_MARKER` in a trajectory's `detail` is cut before the proposer sees it | Ours |
+| Agent self-feedback | If trajectories include `self_feedback` (the agent's own words about the setup after a failing run), it is shown as evidence | Idea from HarnessCompass, arXiv 2608.01918 ("proactive first-person feedback") |
+| Cost by prompt source | If trajectories include `cost_by_source`, the proposer sees where the money goes and is told to change what the harness asks for, never to ask the agent to be brief | Ours |
+
+## Screening an edit (`candidates.py`, `critic.py`)
+
+| What | How | Source |
+|---|---|---|
+| Free structural checks | A candidate must keep the same files, valid JSON, the same setting keys and types, settings within their bounds (ranges or allowed values), the same tools, and no new template placeholders, before anything is spent | RRSI's deterministic checks after the critic |
+| Leakage critic | A regex precheck (credentials, task ids, names from the evaluation data) then an LLM review with reject classes: task specialization, degenerate or safety-weakening changes, grader gaming, undeclared bundling, memory leakage, unbounded work, near-duplicates of rejected ideas, false tool claims. Rejections go back to the proposer for a bounded number of repairs | Adapted from RRSI's critic (google-research/rrsi, Apache-2.0); the instructions are generated from your agent's profile |
+
+## Deciding (`acceptance.py`)
+
+| Rule | How | Source |
+|---|---|---|
+| Noise band | δ = 2 × the bootstrapped standard deviation of the pass-rate change between two runs of the unchanged harness (round 0's trial pairs) | RRSI (`calibrate.py`) |
+| Floor | Never accept a candidate clearly worse than the best so far: S′ ≥ S* − δ | RRSI |
+| Clear gain | A gain beyond δ may cost more, up to 10% plus 44.5% per point gained | RRSI's coding settings (β₀ = 0.10, β₁ = 44.5) |
+| Inside the band | A score change counts for nothing (w_s = 0); the candidate wins only if it is cheaper by more than the cost noise band | RRSI (w_s = 0); the cost band is ours, calibrated the same way from trial-pair costs (`cost_band`, off by default) |
+| Give-up guard | Veto if the share of runs that never submit an answer rises by more than 5 points or its own noise band (calibrated from round 0's trial pairs), whichever is larger | RRSI's domain veto on a rising no-submission rate |
+| Guard cases | Veto if a guard case (one that must keep passing) fails on every trial of the candidate while the incumbent passed it at least once. A single failed trial isn't enough: guards that pass 80–95% of the time would veto about a third of candidates by chance | Ours |
+| Cost only on shared successes | Compare cost only on cases both harnesses solve, so a cheaper failure never counts as a saving (`cost_on_shared_successes`, off by default) | Idea from the Hugging Face / FineEnvs "Multi-harness RL" article: efficiency is rewarded only on correct answers |
+| Near-miss archive | Candidates just inside the noise band are kept and shown to later rounds to build on (`near_miss_fraction`, off by default) | Idea from the Darwin Gödel Machine (arXiv 2505.22954) and Huxley-Gödel Machine (arXiv 2510.21614): the best-scoring variant isn't always the best one to build on |
+| Two-model check | An accepted edit must also hold on a second model's evaluation (`transfer_evaluator`, optional) | Ours, motivated by our finding that a prompt edit evolved for one model did nothing on another |
+| Exploration rule | After N rejected candidates in a row, the next must change a component family none of them touched (`diversity_after`, off by default) | Simplified from RRSI's component novelty |
+
+## Reporting (`report.py`, `pass_k.py`)
+
+| What | How | Source |
+|---|---|---|
+| Held-out comparison | Original and evolved harness on held-out cases, k ≥ 2 trials each | Standard practice; RRSI shows why (an unregularized evolved harness scored 92.8 on its own tasks and 40.3 held out) |
+| Rerun baseline | The evolved harness's single run is compared with the original's best-of-2, estimated without bias from its own k trials. An evolved harness that costs one run should beat that, or you could have just rerun | AI2: Wang et al., "Rethinking the Evaluation of Harness Evolution for Agents", arXiv 2607.12227 |
+| pass@k and pass^k | pass@k = chance at least one of k tries passes; pass^k = chance all k pass (reliability) | Unbiased estimators as in Chen et al. (HumanEval, 2021); pass^k as used in tau-bench (Yao et al., 2024) |
+| Paired bootstrap | Each difference gets a 95% interval from a bootstrap over cases | Standard |
+
+## Running unattended (`loop.py`, `health.py`, `budget.py`, `state.py`)
+
+All ours, from running the optimizer for hours unattended: a hard budget checked before every case;
+provider failures (billing, auth, rate limits) stop the run instead of being scored as failures; an
+unpriced model reply stops the run instead of spending blind; tripwires pause it when measurements look
+broken (a tool failing far above normal, $0 costs, errored runs); a watchdog exits cleanly if it stops
+making progress; an optional smoke stage checks a few cases before real spend.
+
+## Benchmarking the optimizer (`bench.py`)
+
+Known-answer scenarios with a scripted proposer and critic (a real fix must be accepted, a harmful edit and
+a neutral rewording must not), plus open scenarios with the real proposer. Ours. Lesson from the first run:
+measure a planted edit's effect before treating it as a known answer.