@surlymochan/evalid 0.5.1 → 0.5.3

package/CHANGELOG.md

@@ -5,6 +5,22 @@ All notable changes to this project are documented here. Detailed notes live in
 
 ## [Unreleased]
 
+## [0.5.3] - 2026-11-07
+
+### Fixed
+- **Bench openswarm planner timeout raised 300s → 600s (#41)**: the 300s cap was calibrated on DeepSeek, whose planner finished in 87–244s on the 6 pilot instances. With MiniMax M3.1-Flash-Preview at high thinking the planner exceeds 300s on all 6 pilot2 cells: killed mid-work (rc 143) before writing the task file, so the executor never ran and all 6 cells recorded false negatives (wall ~301s). 600s keeps planner + executor inside the CAP_S 1200s cell budget; the executor budget is derived from the remaining time.
+- **Bench openswarm planner prompt tightened to read-only planning (#42)**: the 600s validation cell (pylint-8898) showed the planner spending 13 turns / 23 tools exploring (openswarm lane serve/assign probes, full-disk find for pylint wheels) and getting killed (rc 143) before writing the task file — raising the budget alone cannot fix that. The planner now reads only the skill text and the task, runs no shell command, does not reproduce / explore / list / search, launches no other program, writes the task file + verify command, then stops. The 600s planner timeout from #41 is unchanged. Test evidence: new `bench/swebench/test_planner_prompt.py` (7 tests, stdlib-only via ast so CI can run it without pandas/swebench) + node wrapper `test/bench-planner-prompt.test.js`; bench Python suite 35/35, full `npm test` green.
+
+Acceptance debt, stated plainly: the bench-number acceptance for P1-2 / P1-3 / P2-1 / P2-3 still awaits the MiniMax full-matrix batch (15 instances × 4 arms × reps 3) — no bench numbers are claimed in this release.
+
+## [0.5.2] - 2026-11-07
+
+### Fixed
+- **Bench grading no longer silently zeroes on relative paths (#38)**: `run.py` / `regrade.py` passed `--raw-root` / `--raw` through unresolved, but `swe_env.grade()` runs `git apply <test.patch>` with `cwd=<cell repo>` — with a relative raw root every cell failed grading with `test_patch did not apply` and `run.py` recorded it as an ordinary `resolved: false`. `run.py` now resolves `--out` / `--raw-root` / `--predictions-dir` to absolute paths up front, `regrade.py` resolves `--raw`, and `grade()` writes `grade.json` on the test_patch-apply early return (previously the failure path wrote nothing, which is how this stayed invisible). Found during the 2026-11-06 bench rebuild: the first MiniMax pilot went 4/4 on this error before diagnosis.
+
+### Added
+- **P2-2 15-instance bench set validated (#39, issue #5 P2-2)**: the 3 round-3 additions (sympy-15599, sympy-19040, sympy-24213) have passed the gold-patch check — `harness/validate.py` over all 15 ids in `instances.json` (fa7ea30): 15/15 ok, no errors (`bench-env/validate-15.log`, 2026-11-06 19:35 UTC). `round3_extension.pending_validation` is cleared, the validated ids are recorded, and `test_instances.py` now pins the validated state. The issue #5 P2-2 checkbox is ticked; the bench-number acceptance for P1-2 / P1-3 / P2-1 / P2-3 still awaits the MiniMax full-matrix batch — no bench numbers are claimed here.
+
 ## [0.5.1] - 2026-11-07
 
 ### Added

package/RELEASE_NOTES.md

@@ -1,5 +1,25 @@
 # Release Notes
 
+## 0.5.3 - Bench planner stops timing out and stops wandering
+
+Patch release. Two bench-harness fixes from the MiniMax pilot, both pinned by tests rather than by a live run.
+
+**Openswarm planner timeout 300s → 600s (#41).** The 300s cap was calibrated on DeepSeek, whose planner finished in 87–244s on the 6 pilot instances. With MiniMax M3.1-Flash-Preview at high thinking the planner exceeds 300s on all 6 pilot2 cells: killed mid-work (rc 143) before writing the task file, so the executor never ran and all 6 cells recorded false negatives (wall ~301s). 600s keeps planner + executor inside the CAP_S 1200s cell budget; the executor budget is derived from the remaining time.
+
+**Openswarm planner prompt tightened to read-only planning (#42).** The 600s validation cell (pylint-8898) showed the planner spending 13 turns / 23 tools exploring (openswarm lane serve/assign probes, full-disk find for pylint wheels) and getting killed (rc 143) before writing the task file — raising the budget alone cannot fix that. The planner now reads only the skill text and the task, runs no shell command, does not reproduce / explore / list / search, launches no other program, writes the task file + verify command, then stops. The 600s planner timeout from #41 is unchanged. Test evidence (from the PR): new `bench/swebench/test_planner_prompt.py` (7 tests, stdlib-only via ast so CI can run it without pandas/swebench) + node wrapper `test/bench-planner-prompt.test.js`; bench Python suite 35/35, full `npm test` green.
+
+Acceptance debt, stated plainly: the bench-number acceptance for P1-2, P1-3, P2-1 and P2-3 — the wall-clock/token/resolved lines from issue #5 — still awaits the MiniMax full-matrix batch (15 instances × 4 arms × reps 3), which is running. No bench numbers are claimed in this release.
+
+## 0.5.2 - Bench grades stop silently zeroing; the 15-instance set is validated
+
+Patch release. One bench-harness fix that removes a silent false-negative from grading, plus the P2-2 validation close-out from the issue #5 backlog. Both are pinned by tests and by a validation log, not by a live run.
+
+**Absolute grade paths (#38).** `run.py` / `regrade.py` passed `--raw-root` / `--raw` through unresolved, but `swe_env.grade()` runs `git apply <test.patch>` with `cwd=<cell repo>`. With a relative raw root the patch path did not resolve from the repo cwd, so every cell failed grading with `test_patch did not apply` — and `run.py` recorded it as an ordinary `resolved: false`, a silent false-negative for the entire run. Found during the 2026-11-06 bench rebuild: the first MiniMax pilot went 4/4 on this error before diagnosis. `run.py` now resolves `--out` / `--raw-root` / `--predictions-dir` to absolute paths up front, `regrade.py` resolves `--raw`, and `grade()` also writes `grade.json` on the test_patch-apply early return (previously the failure path wrote nothing — exactly how this stayed invisible). No behavior change for absolute-path invocations. Test evidence (from the PR): `npm run verify` exit 0 (941 pass / 0 fail), including the swebench harness isolation test; the same patch regraded from a fresh tree with absolute paths applies cleanly and grades normally.
+
+**P2-2 15-instance set validated (#39, issue #5 P2-2).** The 3 round-3 additions (sympy__sympy-15599, sympy__sympy-19040, sympy__sympy-24213) have passed the gold-patch check: `harness/validate.py` over all 15 ids in `instances.json` (fa7ea30) reports 15/15 ok:true with no errors (`bench-env/validate-15.log`, mtime 2026-11-06 19:35 UTC, after the instances.json commit). `round3_extension.pending_validation` is cleared to `[]`, the validated ids are recorded, and `test_instances.py` now pins the validated state instead of the pending state. Test evidence (local, this round): `python3 -m unittest test_instances test_diff_filter test_predictions test_reps_report test_validity` — 28 tests, all OK. The issue #5 P2-2 checkbox is ticked.
+
+Acceptance debt, stated plainly: the bench-number acceptance for P1-2, P1-3, P2-1 and P2-3 — the wall-clock/token/resolved lines from issue #5 — still awaits the MiniMax full-matrix batch (15 instances × 4 arms × reps 3), which is running. No bench numbers are claimed in this release.
+
 ## 0.5.1 - Gates stop re-reading unchanged files; bench arms run at high effort
 
 Patch release. Two gate cost improvements from the issue #5 backlog, both pinned by tests rather than by a live run, plus a bench-harness change.

package/docs/launch-dossier.md

@@ -1,6 +1,6 @@
 # evalid Launch Dossier
 
-Generated: 2026-11-06T19:58:36.504Z
+Generated: 2026-11-07T05:49:01.426Z
 Mode: post_publish
 Verdict: conditionally ready: engine and adoption evidence are strong, registry publication remains required for a splash launch
 Launch ready: not yet
@@ -117,7 +117,7 @@ Evidence:
 - Adoption records: 1; status: pass
 - Goal audit: pass; goal: Keep evalid credible as an open-source agent delivery runtime: repo-owned workflow contracts, executable verification gates, durable project context, and honest benchmark evidence.
 - Package evidence: missing; source: registry_json
-- Package name/version: @surlymochan/evalid@0.5.1
+- Package name/version: @surlymochan/evalid@0.5.3
 
 ## Passing Evidence
 

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@surlymochan/evalid",
-  "version": "0.5.1",
+  "version": "0.5.3",
   "description": "Claimed != verified: a runtime verification layer for coding agents. It seals each test run and replays it from a clean snapshot, with a zero-judge gate pipeline (spec clarification, code review, TDD proof) and the xflow CLI.",
   "keywords": [
     "ai-agents",