opencode-harness-kit 0.5.0 → 0.5.1

package/CHANGELOG.md

@@ -12,6 +12,55 @@ is not happening.
 
 ---
 
+## [0.5.1] — 2026-10-21
+
+Published. The tag is what cut it: `git tag v0.5.1 && git push origin main
+v0.5.1`, and the publish job watches for exactly that.
+
+A patch, and it is a patch because nothing a user does differently is in
+here: the grader was stricter than it said, and five tests were reporting
+green with the behaviour they name broken.
+
+### Fixed
+
+- **The grader accepted a field the agent added.** The comparison with the
+  pristine subject looked at the keys the original had, so an agent that made
+  the right change and also added a field of its own was graded correct. The
+  added key is a change like any other, and the grader says so.
+
+- **`isExcluded` did not normalise the path.** The same file is written
+  several ways - "./dist/index.js", "dist/index.js", "src/../dist/index.js" -
+  and the check only caught one spelling. The grader was saved by also
+  checking the resolved path; the exported helper was not, and it is the
+  helper a caller would reach for.
+
+### Test suite
+
+Five tests could not fail, and each was found by breaking the code on
+purpose and watching the test stay green:
+
+- **"a many-file change that reordered the list" did not test reordering.**
+  The fixture put the appended entry first, so a different check caught it
+  and the assertion matched both reasons. The comparison with the original -
+  the check that matters - was never exercised. There are two tests now.
+- **"the pruner changes nothing when there is no profile" could not fail.**
+  The prunable window was empty, and ten DISTINCT outputs all survive
+  anyway. It uses repeated outputs now, with a control case.
+- **"a pruner that throws does not break the turn" never reached the
+  catch.** A string is iterable in JavaScript, so it yielded characters and
+  never threw.
+- **The measurement test did not assert that the measurement measured
+  anything.** It ran behind a gate that can never fail and only matched the
+  header, so a pruner that measured nothing passed.
+- **The python-fallback test never removed the sqlite3 CLI**, so the
+  fallback it was about never ran on a runner that has it.
+
+Also fixed: a lane loop that could iterate zero times; two escape hatches
+that made a test assert nothing at all; a git-root test that used the
+checkout and so failed in a source tarball; seventeen temporary directories
+leaked a run; and a profile index read with a fallback that made a missing
+key mean "no entries" rather than a failure.
+
 ## [0.5.0] — 2026-10-21
 
 A MINOR release, and the reason is the pruner: the rule that decides what

package/CONTRIBUTING.md

@@ -69,6 +69,13 @@ config, run the layers individually and say so in the PR.
 not part of any check, and a PR does not need it. `npm run measure` does not
 spawn an agent either — it queries the install database read-only.
 
+The nightly workflow needs a provider to measure with, and the provider key
+is a **secret** — which a job-level `if` cannot read. The flag is therefore
+a repository **variable**: set `HAS_OPENCODE_KEY` to `true` when
+`OPENCODE_API_KEY` is there, and the job runs. Without the variable the job
+is skipped, not failed, because a schedule that is always red is a schedule
+nobody reads.
+
 `eval/models.mjs` spawns real agents too, one line each, against the live
 tier: it is what the routing table in `docs/ROUTING.md` is measured with,
 and it costs a request per model per rep. It is not part of any check

package/README.md

@@ -830,7 +830,7 @@ short never grows the history far enough for pruning to matter. So there is
 a second suite with one task that touches twelve files and asks a question
 about the code that consumes what it changed — and on it, with the pruner's
 rule fixed (see below), the kit's arm makes the same number of tool calls
-as the arm without the kit and takes 1.2x the wall time.
+as the arm without the kit (47 against 46) and takes 1.2x the wall time.
 
 **The pruner used to make the agent re-read.** The first run of that task
 said the kit was much worse: 86 tool calls against 43 and 282,000 input

package/eval/RESULTS.md

@@ -221,11 +221,11 @@ pristine subject file by file, and the answer is graded by citation.
 | metric (median of 2 runs per arm) | kit on | kit off |
 | --- | --- | --- |
 | **correct** | **2/2** | **2/2** |
-| tool calls | 51 | 39.5 |
-| read calls | 18.5 | 13.5 |
-| input tokens | 84,325 | 41,291 |
-| final context | 75,705 | 45,647 |
-| wall seconds | 123.5 | 104.8 |
+| tool calls | 47 | 46 |
+| read calls | 12 | 18 |
+| input tokens | 78,280 | 37,171 |
+| final context | 68,794 | 42,732 |
+| wall seconds | 121.4 | 104.7 |
 
 ### What the long task found, and what it cost to fix
 
@@ -259,8 +259,8 @@ The context is bigger and the work is not repeated. That is the trade the
 kit was making badly, and it was making it without saying.
 
 **What is left is the instructions, not the pruner.** With the rule on, the
-kit's arm makes the same number of tool calls as the arm without the kit
-(51 against 39.5, against 67.5 before the fix) and takes 1.2x the wall time.
+kit's arm makes the SAME number of tool calls as the arm without the kit -
+47 against 46, against 67.5 before the fix - and takes 1.2x the wall time.
 The remaining cost is the fixed overhead of carrying the kit's rules, and
 that is the trade the README describes.
 

package/eval/verify.mjs

@@ -180,14 +180,31 @@ export function resolveInTarget(target, ref) {
  * @returns {boolean}
  */
 export function isExcluded(file, exclude = []) {
-  const base = file.split("/").pop()
+  // The path is NORMALISED, because the same file is written several ways:
+  // "./dist/index.js", "dist/index.js" and "src/../dist/index.js" are one
+  // file, and a check that only catches one spelling is not a check.
+  const norm = String(file ?? "")
+    .replace(/\\/g, "/")
+    .replace(/^\.\//, "")
+    .replace(/^\.\//g, "")
+  const parts = []
+  for (const seg of norm.split("/")) {
+    if (!seg || seg === ".") continue
+    if (seg === "..") {
+      parts.pop()
+      continue
+    }
+    parts.push(seg)
+  }
+  const clean = parts.join("/")
+  const base = parts[parts.length - 1] ?? ""
   return exclude.some((x) => {
     if (!x) return false
-    if (x.endsWith("/")) return file.startsWith(x)
+    if (x.endsWith("/")) return clean.startsWith(x)
     // A file NAME excludes that file wherever it is: the suite is excluded as
     // "tasks-self.json", and the citation is "eval/tasks-self.json".
-    if (file === x || base === x) return true
-    return file.startsWith(`${x}/`)
+    if (clean === x || base === x) return true
+    return clean.startsWith(`${x}/`)
   })
 }
 
@@ -353,6 +370,13 @@ export function gradeMutationMany(runDir, spec, { pristine } = {}) {
         return { ok: false, reason: `${file}: ${key} changed, and only ${spec.field} was asked for` }
       }
     }
+    // And a key the agent ADDED is a change too. Only the keys the original
+    // had were compared, so an agent that appended the right entry and also
+    // added a field of its own was graded correct.
+    for (const key of Object.keys(data)) {
+      if (key in orig) continue
+      return { ok: false, reason: `${file}: ${key} was added, and only ${spec.field} was asked for` }
+    }
     if (JSON.stringify(arr) !== JSON.stringify(origArr)) changed.push(file)
   }
   return { ok: true, changed }

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "opencode-harness-kit",
-  "version": "0.5.0",
+  "version": "0.5.1",
   "description": "Harness kit for OpenCode: capability profiles, a fleet of job lanes with one free model each, role-based reasoning effort, context pruning, and measured baselines.",
   "license": "MIT",
   "homepage": "https://github.com/pham-canh/space-harness#readme",