@metaharness/darwin 0.2.2 → 0.2.4

package/CHANGELOG.md

@@ -2,6 +2,14 @@
 
 All notable changes to this package. Dates UTC.
 
+## 0.2.4 — 2026-06-19
+
+- New result (ADR-151): swapping the cheap base deepseek-V3 -> deepseek-v4-pro nearly doubles the repair floor **15.3% -> 29.3%** [24.5,34.7] on full SWE-bench Lite (same harness). Falsifies "paradigm exhausted regardless of model IQ."
+
+## 0.2.3 — 2026-06-19
+
+- Add `LEARNINGS.md` — the measured findings distilled into harness defaults (repair=2x lever, cost-routing, Barbarian&Scholar tiering, format-contract, batch-eval discipline, capability floor).
+
 ## 0.2.2 — 2026-06-19
 
 - **Docs: full SWE-bench Lite (300) evidence ladder** now in the description + README, all official `swebench` Docker harness, batch-verified:

package/LEARNINGS.md

@@ -0,0 +1,63 @@
+# What the benchmarks taught us → harness defaults
+
+Empirical findings from the full SWE-bench Lite (300) arc (official `swebench` Docker harness,
+batch-verified — see `bench/results/RESULTS.md`). These are the *measured* reasons behind the
+recommended harness patterns. The headline: **the harness, not the model, is the dominant lever.**
+
+## 1. Closed-loop repair (test feedback) is the #1 lever — ~2× for free
+- open-loop single-shot: **7.7%** → + closed-loop repair (run the failing tests, feed the
+  traceback back, retry ≤3): **15.3%** — on the *same cheap model*, ~$0.01/instance.
+- **Recommendation:** make iteration against ground-truth (compiler/tests) first-class. A model
+  that can *see why it failed* beats a smarter model that can't. Prefer `retryPolicy` configs that
+  consume real failure signal over blind retries.
+
+## 2. Localization fixes retrieval, not emission — beware the "emission wall"
+- LLM file-localization lifted gold-file recall **+15pp** but resolve-rate stayed flat (8.0%).
+  The bottleneck was *writing a valid patch*, not *finding the file*.
+- **Recommendation:** measure where you actually lose (selection vs emission) before optimizing
+  retrieval. Don't assume better context = better output.
+
+## 3. Format contract + fit-in-context unblocks weak/local models (0 → 13/25 applied)
+- A small local model emitted prose summaries instead of edits until the harness (a) served enough
+  context window, (b) carried the search/replace **format contract in a system message + worked
+  example**, and (c) **shrank per-file context to fit the window** (truncation silently dropped the
+  instruction). Apply-rate went 0 → ~50%.
+- **Recommendation:** put the output-format contract in a *system* role with an example; size the
+  prompt to the model's real context; never let truncation eat the instruction.
+
+## 4. Cheap-first + cost-aware routing — 31× cheaper per resolve
+- Router probe: `pareto-code`→deepseek-v4-pro resolved at **$0.21/resolve** vs `fusion`→opus-4.8 at
+  **$6.57/resolve** — same task, 31× cost gap for +1 resolve.
+- **Recommendation:** default to the cheapest model that clears the task; reserve frontier models for
+  measured capability gaps. Track **$/resolve**, not just resolve-rate.
+
+## 5. Barbarian & Scholar — tier the models, escalate only the residual (33.3% at ~6× less cost)
+- Cheap base (deepseek + repair) banks the easy 46/300; a frontier "Scholar" (sonnet-4 + repair)
+  escalated **only to the 254 it failed** cracks 55 more → **100/300 = 33.3%**, blended
+  ~$0.34/instance vs ~$2 to run frontier on all 300.
+- **Recommendation:** two-tier orchestration — cheap sweep, then frontier on the residual — is far
+  more cost-efficient than one strong model everywhere (you'd waste 5/6 of frontier spend re-solving
+  what cheap already gets).
+
+## 6. The repair lift is model-bound below a capability floor (~14B)
+- Repair did nothing for a 7B (4%→4%) but lifted a 14B (8%→12%) and doubled a hosted model. The loop
+  needs the model to *occasionally* produce a correct-ish patch to converge toward.
+- **Recommendation:** don't expect harness scaffolding to rescue a model below the task's reasoning
+  floor; pick the smallest model *above* it, then let the harness multiply it.
+
+## 7. Methodology: only batch-eval on final predictions is authoritative
+- In-loop "resolved" counters drifted from clean batch eval by 1.5–5× (both directions — flaky
+  passes over-count; Docker-hang false-negatives under-count). Every reported number here is a
+  fresh batch eval on the final saved predictions.
+- **Recommendation:** never report the in-loop signal; re-evaluate the artifact you'd actually ship.
+
+## 8. Engineering robustness (or your run lies to you)
+- Concurrency clones rate-limit (6-wide anon GitHub clones → 63 fetch failures): **cap at 2–3**,
+  retry-with-backoff, free each clone. One instance (`psf__requests-2317`) reliably hangs Docker
+  past timeout → known-flaky exclusion (`bench/swebench/KNOWN_FLAKY.md`). Watch for wedged containers.
+
+---
+
+Verdict: this paradigm (localize + search/replace + repair + tiered escalation) tops out ~33% on
+SWE-bench Lite with a cheap base. The 65–88% agentic-SOTA tier needs a **multi-step autonomous agent**
+(bash, dir-navigation, long-horizon discovery) — an architecture change, not more knob-tuning.

package/README.md

@@ -273,7 +273,7 @@ substrate is **no longer deferred** — it shipped (ADR-106→141) and now runs
 
 ### Canonical SWE-bench Lite (real, official harness — ADR-142–149)
 
-> Full reproducible evidence + per-run provenance: [`bench/results/RESULTS.md`](bench/results/RESULTS.md) · known-flaky exclusions: [`bench/swebench/KNOWN_FLAKY.md`](bench/swebench/KNOWN_FLAKY.md)
+> Full reproducible evidence: [`bench/results/RESULTS.md`](bench/results/RESULTS.md) · measured best-practices: [`LEARNINGS.md`](LEARNINGS.md) · known-flaky exclusions: [`bench/swebench/KNOWN_FLAKY.md`](bench/swebench/KNOWN_FLAKY.md)
 
 Run on the **full 300** SWE-bench Lite (test) instances, scored by the **official
 `swebench` Docker harness** — no cherry-picking, tight CIs. Solver = relevance-ranked
@@ -284,6 +284,7 @@ context + symbol-aware localization + search/replace patch, `deepseek-chat`, ~$0
 | baseline (open-loop, single-shot) | 23/300 = **7.7%** | [5.2, 11.2] | 144 |
 | + LLM localization | 24/300 = **8.0%** | [5.4, 11.6] | 146 |
 | **+ closed-loop repair (test-feedback, ≤3)** | 46/300 = **15.3%** | **[11.7, 19.8]** | 149 |
+| **+ swap base → deepseek-v4-pro (cheap)** | 88/300 = **29.3%** | **[24.5, 34.7]** | 151 |
 | **+ Barbarian&Scholar hybrid (cheap+frontier tail)** | 100/300 = **33.3%** | **[28.2, 38.8]** | 148 |
 
 The closed-loop repair loop **~doubles** the resolve-rate (7.7% → 15.3%) on the *same cheap model*

package/package.json

@@ -1,7 +1,7 @@
 {
   "name": "@metaharness/darwin",
-  "version": "0.2.2",
-  "description": "An LLM supercharger and cost optimizer: freeze your model, evolve the harness around it. Measured on full SWE-bench Lite (300, official swebench Docker): open-loop 7.7% -> +repair 15.3% -> +Barbarian&Scholar hybrid 33.3%, at ~$0.01-$0.34/instance (vs $1-20 for frontier agents). Mutate->sandbox->score->archive. Dependency-free (Node built-ins).",
+  "version": "0.2.4",
+  "description": "An LLM supercharger and cost optimizer: freeze the model class, evolve the harness. Measured on full SWE-bench Lite (300, official swebench Docker): cheap-model repair 15.3% (deepseek-V3) -> 29.3% (deepseek-v4-pro) -> 33.3% Barbarian&Scholar hybrid, ~$0.01-$0.34/instance (vs $1-20 for frontier agents). The harness, not the model, is the lever. Dependency-free (Node built-ins).",
   "type": "module",
   "main": "./dist/index.js",
   "types": "./dist/index.d.ts",
@@ -19,7 +19,8 @@
     "README.md",
     "SECURITY.md",
     "LICENSE",
-    "CHANGELOG.md"
+    "CHANGELOG.md",
+    "LEARNINGS.md"
   ],
   "scripts": {
     "build": "tsc",