opencode-harness-kit 0.4.6 → 0.5.1

package/CHANGELOG.md

@@ -12,6 +12,126 @@ is not happening.
 
 ---
 
+## [0.5.1] — 2026-10-21
+
+Published. The tag is what cut it: `git tag v0.5.1 && git push origin main
+v0.5.1`, and the publish job watches for exactly that.
+
+A patch, and it is a patch because nothing a user does differently is in
+here: the grader was stricter than it said, and five tests were reporting
+green with the behaviour they name broken.
+
+### Fixed
+
+- **The grader accepted a field the agent added.** The comparison with the
+  pristine subject looked at the keys the original had, so an agent that made
+  the right change and also added a field of its own was graded correct. The
+  added key is a change like any other, and the grader says so.
+
+- **`isExcluded` did not normalise the path.** The same file is written
+  several ways - "./dist/index.js", "dist/index.js", "src/../dist/index.js" -
+  and the check only caught one spelling. The grader was saved by also
+  checking the resolved path; the exported helper was not, and it is the
+  helper a caller would reach for.
+
+### Test suite
+
+Five tests could not fail, and each was found by breaking the code on
+purpose and watching the test stay green:
+
+- **"a many-file change that reordered the list" did not test reordering.**
+  The fixture put the appended entry first, so a different check caught it
+  and the assertion matched both reasons. The comparison with the original -
+  the check that matters - was never exercised. There are two tests now.
+- **"the pruner changes nothing when there is no profile" could not fail.**
+  The prunable window was empty, and ten DISTINCT outputs all survive
+  anyway. It uses repeated outputs now, with a control case.
+- **"a pruner that throws does not break the turn" never reached the
+  catch.** A string is iterable in JavaScript, so it yielded characters and
+  never threw.
+- **The measurement test did not assert that the measurement measured
+  anything.** It ran behind a gate that can never fail and only matched the
+  header, so a pruner that measured nothing passed.
+- **The python-fallback test never removed the sqlite3 CLI**, so the
+  fallback it was about never ran on a runner that has it.
+
+Also fixed: a lane loop that could iterate zero times; two escape hatches
+that made a test assert nothing at all; a git-root test that used the
+checkout and so failed in a source tarball; seventeen temporary directories
+leaked a run; and a profile index read with a fallback that made a missing
+key mean "no entries" rather than a failure.
+
+## [0.5.0] — 2026-10-21
+
+A MINOR release, and the reason is the pruner: the rule that decides what
+survives changed, so the default behaviour of the kit changed. A patch
+would have been a lie about what is in the box.
+
+Published. The tag is what cut it: `git tag v0.5.0 && git push origin main
+v0.5.0`, and the publish job watches for exactly that.
+
+This release is the one where the kit measured itself properly, and what it
+found was a defect in the kit: the pruner was making the agent read the
+same files again.
+
+### Added
+
+- **A long task for the A/B**, and a grader that can check one. The short
+  suite is two-line questions answered in one or two reads, and on those the
+  kit is a cost - over five times the input tokens and a tie on correctness.
+  That is the wrong task for the kit's central claim, which is that the
+  pruner pays for itself on a LONG session: a short session never grows the
+  history far enough for pruning to matter. The new task touches twelve files
+  and asks a question about the code that consumes what it changed, so the
+  conversation is long because the work is long.
+
+  Grading it needed a new kind of check. A grader that looks at one file
+  cannot say whether the other eleven were touched, so the check is a
+  COMPARISON with the pristine subject: every file in the run copy must be
+  the original with exactly one entry appended, and the file for the model
+  being appended must be untouched - a model cannot fall back to itself.
+
+- **The process, measured.** The kit claims a reviewer that reviews and does
+  not edit. A task that invites the agent to fix what it finds, graded on
+  the finding being cited AND the file being untouched, gives 2/2 against
+  2/2: the kit's reviewer lane and the default agent behaved the same. The
+  lane is not what made them behave, and it is soft - a task that says "fix
+  it" overrides it. Both are written down, because a claim that is not
+  measured is a claim.
+
+- **A public TypeScript subject.** The pilot ran against a private
+  repository, which means nobody else could repeat it. The subject is now
+  `typesense/typesense-js` at a pinned commit: Apache-2.0, TypeScript source
+  under `src/Typesense/` and the COMPILED JavaScript committed next to it in
+  `lib/` and again in the unminified bundle. Every answer occurs three times,
+  and the bundle keeps the original module banners, so a search for a source
+  path lands in the build output. The harness clones it itself:
+  `eval/prepare.mjs --clone <url> --ref <sha>`.
+
+### Changed
+
+- **The pruner keeps the newest copy of each DISTINCT output.** It used to
+  keep the last six tool outputs and stub everything older, which is a window
+  on RECENCY: after six more calls, the only copy of a file the agent had
+  read was gone, and the agent read it again. The long task measured it -
+  seventeen re-reads, five times the input tokens, and the same final
+  context. The newest copy of each distinct output now survives however old
+  it is, and only the repeats are stubbed. Same subject, same task, same
+  model: **-69% input tokens and -60% wall time**, at the cost of a bigger
+  context. It is a setting, `prune.keepDistinctOutputs`, so the old
+  behaviour is one profile edit away.
+
+### Fixed
+
+- **The nightly A/B failed every night** because the repository has no
+  provider secret. The job is skipped when the key is missing, because a
+  schedule that is always red is a schedule nobody reads.
+
+- **`oc-harness uninstall` and the `restore` command** now keep the dated
+  backups in one place, and the prompt that asks "overwrite the config?"
+  says where the backup goes. It said ".bak", and the only thing that wrote
+  a `.bak` was `restore`.
+
 ## [0.4.6] — 2026-10-21
 
 Published. The tag is what cut it: `git tag v0.4.6 && git push origin main

package/CONTRIBUTING.md

@@ -69,6 +69,13 @@ config, run the layers individually and say so in the PR.
 not part of any check, and a PR does not need it. `npm run measure` does not
 spawn an agent either — it queries the install database read-only.
 
+The nightly workflow needs a provider to measure with, and the provider key
+is a **secret** — which a job-level `if` cannot read. The flag is therefore
+a repository **variable**: set `HAS_OPENCODE_KEY` to `true` when
+`OPENCODE_API_KEY` is there, and the job runs. Without the variable the job
+is skipped, not failed, because a schedule that is always red is a schedule
+nobody reads.
+
 `eval/models.mjs` spawns real agents too, one line each, against the live
 tier: it is what the routing table in `docs/ROUTING.md` is measured with,
 and it costs a request per model per rep. It is not part of any check

package/README.md

@@ -539,6 +539,16 @@ the same messages with and without. On one install, on 2026-10-20:
 sessions, because it is measured over the ones this machine happened to
 have - so run the command for the number on yours.
 
+**What is kept is the newest copy of each DISTINCT output.** The window on
+recent calls was a window on *recency*, and a long session walks out of it:
+after six more calls, the only copy of a file the agent had read was gone,
+and the agent read it again. On the long-task A/B that was seventeen
+re-reads and five times the input tokens. The newest copy of each distinct
+output now survives however old it is, and only the repeats are stubbed -
+**-69% input tokens and -60% wall time** on that task, at the cost of a
+bigger context. It is a setting: `prune.keepDistinctOutputs` in the profile,
+`false` to get the old behaviour back.
+
 ## Memory
 
 The kit does not implement memory. A local vector store with embeddings, a
@@ -814,11 +824,32 @@ next, which is the shape some of the kit's answers took. Re-scoring the
 same 120 runs with the fixed grader gives 34/40 against 30/40 on the lookup
 tasks - the kit ahead, where the old grader had it behind. The full account is in [eval/RESULTS.md](eval/RESULTS.md).
 
-What the kit buys is in the two places that suite cannot see: **long
-sessions**, where the pruner removes 65-80% of the context depending on the
-sessions it is measured over, and **process** -
-a reviewer that reviews and does not edit, a verify that runs the checks and
-does not fix them, a publish decision that belongs to a human.
+**On a long task the pruner is where the kit pays.** The suite above is
+made of two-line questions answered in one or two reads, and a session that
+short never grows the history far enough for pruning to matter. So there is
+a second suite with one task that touches twelve files and asks a question
+about the code that consumes what it changed — and on it, with the pruner's
+rule fixed (see below), the kit's arm makes the same number of tool calls
+as the arm without the kit (47 against 46) and takes 1.2x the wall time.
+
+**The pruner used to make the agent re-read.** The first run of that task
+said the kit was much worse: 86 tool calls against 43 and 282,000 input
+tokens against 51,000, for the same final context. Seventeen of the reads
+were re-reads. The pruner keeps the last six tool outputs and stubs
+everything older — a window on *recency* — so after six more calls the only
+copy of a file the agent had read was gone, and the agent read it again. The
+newest copy of each **distinct** output now survives however old it is, and
+only the repeats are stubbed. Same subject, same task, same model: **-69%
+input tokens and -60% wall time**, at the cost of a bigger context.
+
+**The process is measured too, and the result is not flattering.** The kit
+claims a reviewer that reviews and does not edit. A task that invites the
+agent to fix what it finds, graded on the finding being cited AND the file
+being untouched, gives **2/2 against 2/2**: the kit's reviewer lane and the
+default agent behaved the same. The lane is not what made them behave, and
+it is soft — a task that says "fix it" overrides it. Both are in
+[eval/RESULTS.md](eval/RESULTS.md), because a claim that is not measured is
+a claim.
 
 Whether it is worth it depends on whether the work is long. A repository of
 one-file lookups should not install this. The full table is in

package/dist/index.js

@@ -670,7 +670,8 @@ function normalize(p) {
       enabled: p.prune?.enabled ?? false,
       keepRecentToolCalls: p.prune?.keepRecentToolCalls ?? 6,
       keepRecentTurns: p.prune?.keepRecentTurns ?? 3,
-      keepUserMessages: p.prune?.keepUserMessages ?? true
+      keepUserMessages: p.prune?.keepUserMessages ?? true,
+      keepDistinctOutputs: p.prune?.keepDistinctOutputs ?? true
     },
     fallback: p.fallback ?? []
   };
@@ -942,6 +943,9 @@ function validate(p, path = "<memory>") {
     if (prune.keepUserMessages !== void 0 && typeof prune.keepUserMessages !== "boolean") {
       throw new ProfileError(`${where}: prune.keepUserMessages must be a boolean`);
     }
+    if (prune.keepDistinctOutputs !== void 0 && typeof prune.keepDistinctOutputs !== "boolean") {
+      throw new ProfileError(`${where}: prune.keepDistinctOutputs must be a boolean`);
+    }
     if (prune.enabled !== void 0 && typeof prune.enabled !== "boolean") {
       throw new ProfileError(`${where}: prune.enabled must be a boolean`);
     }
@@ -1166,6 +1170,17 @@ function activeProfileId(worktree) {
   return null;
 }
 var REASONING_STUB = "[reasoning] dropped from context - it was from an earlier turn and is stale.";
+function toolSignature(part) {
+  const tool = part?.state?.tool ?? part?.tool;
+  if (!tool) return null;
+  let input;
+  try {
+    input = JSON.stringify(part?.state?.input ?? {});
+  } catch {
+    return null;
+  }
+  return `${tool} ${input}`;
+}
 function sizeOf(part) {
   if (typeof part?.text === "string") return part.text.length;
   if (typeof part?.state?.output === "string") return part.state.output.length;
@@ -1197,6 +1212,7 @@ var PrunePlugin = async ({ worktree, directory }) => {
     keepCalls: 0,
     keepTurns: 0,
     keepUsers: true,
+    keepDistinct: false,
     threshold: 0
   };
   let cached = null;
@@ -1215,6 +1231,7 @@ var PrunePlugin = async ({ worktree, directory }) => {
       keepCalls: profile.prune.keepRecentToolCalls ?? 6,
       keepTurns: profile.prune.keepRecentTurns ?? 3,
       keepUsers: profile.prune.keepUserMessages ?? true,
+      keepDistinct: profile.prune.keepDistinctOutputs ?? true,
       threshold: pruneThreshold(profile)
     };
     return cached;
@@ -1222,7 +1239,7 @@ var PrunePlugin = async ({ worktree, directory }) => {
   return {
     "experimental.chat.messages.transform": async (_input, output) => {
       try {
-        const { profile, keepCalls, keepTurns, keepUsers, threshold } = resolve4();
+        const { profile, keepCalls, keepTurns, keepUsers, keepDistinct, threshold } = resolve4();
         if (!profile) return;
         const messages = output.messages;
         if (!Array.isArray(messages) || messages.length === 0) return;
@@ -1239,6 +1256,21 @@ var PrunePlugin = async ({ worktree, directory }) => {
             seen++;
           }
         }
+        if (keepDistinct) {
+          const seenSignatures = /* @__PURE__ */ new Set();
+          for (let i = messages.length - 1; i >= 0; i--) {
+            const msg = messages[i];
+            if (!msg?.parts) continue;
+            for (const part of msg.parts) {
+              if (part?.type !== "tool") continue;
+              if (part.state?.status !== "completed") continue;
+              const sig = toolSignature(part);
+              if (!sig || seenSignatures.has(sig)) continue;
+              seenSignatures.add(sig);
+              keepToolMessageIds.add(msg.info?.id ?? String(i));
+            }
+          }
+        }
         const stats = {
           stubbed: 0,
           kept: 0,

package/dist/profile.d.ts

@@ -54,6 +54,7 @@ export type Profile = {
         keepRecentToolCalls?: number;
         keepRecentTurns?: number;
         keepUserMessages?: boolean;
+        keepDistinctOutputs?: boolean;
     };
     fallback?: string[];
 };
@@ -86,7 +87,8 @@ export type RetentionRecord = {
  * @property {Record<string, EffortRequest>} [roles]
  * @property {Record<string, RoleModel>} [roleModels]
  * @property {{ enabled?: boolean, keepRecentToolCalls?: number,
- *             keepRecentTurns?: number, keepUserMessages?: boolean }} [prune]
+ *             keepRecentTurns?: number, keepUserMessages?: boolean,
+ *             keepDistinctOutputs?: boolean }} [prune]
  * @property {string[]} [fallback]
  * @typedef {{ retention: Retention, note?: string, verified?: string, source?: string }} RetentionRecord
  */

package/docs/KNOWN-LIMITS.md

@@ -175,6 +175,11 @@ writes it to the project root, and reading `process.cwd()` alone reported
 
 ## The free tier cannot read a PDF, an audio file or a video
 
+**Re-checked 2026-10-21, against the live tier, and both still hold.** A
+PDF attached to a prompt is refused by the provider with the message below,
+and an `.m4a` is refused by opencode with `Cannot read binary file`. The
+model then answers as if it had heard nothing.
+
 Three capabilities the catalogue advertises are not reachable, and no lane
 claims them:
 
@@ -342,9 +347,19 @@ does; if it does not, `prune.ts` is doing the work anyway.
 
 `prune.ts` keeps pointers rather than bodies. Full tool output is not deleted
 from the session record, and OpenCode already writes over-cap output to its own
-truncation directory, so re-reading is possible. But "re-run the command to see
-it again" is more expensive than "scroll up". The stub is one line precisely so
-an agent can decide whether it is worth it, and the stub says so.
+truncation directory, so re-reading is possible. The stub is one line precisely
+so an agent can decide whether it is worth it, and the stub says so.
+
+**But the newest copy of each DISTINCT output survives.** The window on
+RECENT calls was a window on recency, and a long session walks out of it: after
+six more tool calls, the only copy of a file the agent had read was gone, and
+the agent read it again. The long-task A/B measured seventeen re-reads, five
+times the input tokens and the same final context. The rule now keeps the
+newest copy of each distinct output however old it is, and stubs only the
+REPEATS of it. The context grows by the size of the distinct outputs, and the
+re-reads stop: **-69% input tokens and -60% wall time** on that task, at the
+cost of a bigger context. It is a setting - `prune.keepDistinctOutputs` - and
+turning it off brings the old behaviour back.
 
 A `savedChars` of 0 in `prune.jsonl` is not a second, redundant pass. It means
 `bytesAfter` did not come out below `bytesBefore` for that turn, which happens
@@ -361,6 +376,22 @@ possible. Treat the ratio as a guide, not an exact count. It does import the
 real allowlist rather than keeping a copy, so the list cannot drift from what is
 actually shipped.
 
+## A lane is a default, not a boundary
+
+The kit's reviewer prompt says: *"You review. You never fix, because a
+reviewer who edits stops being a reviewer."* That is an instruction, and an
+instruction is only as strong as the task it sits next to.
+
+It was measured. A task that says "review this and FIX it if it is wrong"
+makes the reviewer lane edit the file, and makes the default agent edit it
+too - both arms scored zero on "the finding was reported AND the file was
+untouched". A task that says "tell me what needs to change" makes both arms
+report and neither edit, so the lane made no difference there either.
+
+So the lane holds a line when the task does not push against it, and the
+kit's own loop keeps that line by never asking a reviewer to fix anything.
+If you write a review task that says "fix it", the reviewer will fix it.
+
 ## What this kit does not do
 
 It does not implement memory persistence, worktree orchestration, a context