opencode-harness-kit 0.6.3 → 0.7.0

package/CHANGELOG.md

@@ -12,6 +12,294 @@ is not happening.
 
 ---
 
+## [0.7.0] — 2026-10-25
+
+`oc-harness run` now stops for the reasons it already claimed to stop for,
+the read-only lanes no longer have a write-capable shell, and the token budget
+counts the context a request reads instead of only its uncached part.
+
+### Fixed
+
+- **A denied tool call was fed back as another turn, and the retry spent the
+  window.** `experimental.continue_loop_on_deny` was `true`, so opencode
+  handed the guard's refusal back to the model and continued the loop. The
+  measured session made 138 requests and 7,559,016 context tokens in 29
+  minutes on six `shared token window` denials, each retry re-sending the
+  whole prompt. The generated config now sets it to `false`, and the guard
+  blocks a single request that re-sends more than `OC_HARNESS_TURN_STOP`
+  (default 300k) with "do not retry".
+- **The in-event shared-window stop did not refuse anything.** When the
+  `step-finish` sum crossed `OC_HARNESS_BUDGET_WINDOW_STOP`, the guard set
+  `windowSpent` but never set `windowWhy`, so `sharedWindowSpent()` returned
+  an empty reason and every tool call was allowed. The reason is set where
+  the sum crosses, and the stop is written to the ledger with
+  `recordStop()` so a new session cannot outlive the provider's window.
+- **A corrupt ledger locked the machine for six hours, or vanished.** It is
+  now moved to `budget.json.corrupt` and the window starts empty; the broken
+  bytes are kept as evidence. `recordSpend()` and `recordStop()` repair
+  before writing instead of reading the corruption as an empty window.
+- **The bundled ECC subagents were loaded on every request.** Their
+  descriptions came from `prompts/agents/*.txt`, were not lanes, and so were
+  not covered by the stale-lane rule. They are removed from the generated
+  agent map when that is the prompt, and named in the config header; a
+  hand-written agent that reuses that prompt path is removed like the
+  bundle's own, while any other prompt is left alone.
+- **The token budget undercounted a cached session five to forty times, so it
+  never fired.** A `step-finish` part reports `input`, `cache.read`,
+  `cache.write`, `output` and `reasoning`. The guard summed `input` alone -
+  the part of the prompt that was not cached - while the cached prefix, which
+  is what a long conversation actually re-reads, was ignored. One measured
+  stream reported 844 uncached tokens against 36,480 cached ones on the same
+  step, then 391 against 9,088. The 2M warn and 5M stop were therefore five
+  to forty times the real spend, and the provider's window ran out before
+  either fired. `requestTokens()` in `src/budget.js` now counts `input + cache.read +
+  cache.write` for both the session total and the shared-window ledger; the
+  toasts and refusals say "context tokens" because that is what they count.
+- **The A/B's `--budget` had the same undercount.** The suite cap summed
+  `inputTokens`, so 120 runs reported about 1.3M of spend while their streams
+  carried 7.1M. `parseStream` now also returns `contextTokens`, the cap and the
+  report use it, and `--rescore` re-reads the old streams to put the corrected
+  number on them. `inputTokens` is still recorded so a table can be compared
+  with the ones written before this.
+- **The test suite wrote to the real window ledger.** `guard-hook.test.js`
+  exercised the guard with the default `OC_HARNESS_BUDGET_FILE`, so every test
+  run added its 1,500 and 600 tokens to
+  `~/.local/state/opencode-harness-kit/budget.json` - the live ledger showed
+  twelve test entries and nothing else. The tests now point the ledger at
+  their own temporary directory.
+- **`--rescore` graded a timeout as a wrong answer.** A run that hit the
+  ceiling has a stream that stops mid-sentence, which reads exactly like an
+  answer that cited nothing; only the sidecar knows which it was. Four runs
+  that timed out at 12 steps were re-scored as four wrong answers, and the
+  table then showed 18/23 where the honest count is 18/19.
+- **A resume crashed the report on a sidecar from an older version.** The
+  re-score reads `escaped`, `tools` and `text` from every result; a sidecar
+  written before those fields existed threw a `TypeError` after every run had
+  already been paid for. Missing fields are read as empty, and the corrected
+  token count is written back so a second resume does not depend on the stream
+  still being there.
+- **The A/B's preflight said "ok" to a run that had failed.** It checked only
+  the timeout and the quota, so a run that reported an error event and exited
+  in three seconds passed. The first suite with that preflight spent eighteen
+  runs learning that every kit-off arm had failed with
+  `ProviderModelNotFoundError`, scored as "no output, not scored". It now
+  reads the stream, and any error event stops the suite with exit 2 and the
+  provider's own message. An exit 0 with zero steps and no text is the same
+  shape - a spent route - and is refused the same way.
+- **The kit-off arm could not resolve the model it was told to use.** opencode
+  keeps its model catalogue in `~/.cache/opencode/models.json`, and the arm's
+  temporary `HOME` had no cache, so a bare `model:` string was a model the
+  installation had never heard of. The cache is opencode's, not the kit's, so
+  it is symlinked into the temporary home the same way `~/.local/share/opencode`
+  already was.
+- **An error in a measured run was counted as a wrong answer.** A run that
+  produced no tool call and no text, but did report an error, now stops the
+  suite at once with that error instead of waiting for `--fail-fast` and then
+  scoring the refusal.
+- **The lane counter's "no telemetry" test failed on a machine that had one
+  two directories up.** The lookup walks up to the project root by design, so
+  `/tmp/.space/telemetry.jsonl` - written by any `opencode run` started in
+  `/tmp` - made it find a file and fail for a reason that was not the code's.
+  The test now runs twelve levels down, past the walk-up's ceiling, so nothing
+  above its own tree is looked at; the walk-up itself is unchanged.
+- **The A/B was governed by the kit's own budget.** The kit-on arm runs on the
+  real config, so the guard's session and window thresholds applied to it: 120
+  runs spend more than the window's stop, and from the run that crossed it the
+  guard refused tool calls - every later run would be scored as a wrong answer
+  for a reason the suite is not measuring. The suite now sets all four
+  thresholds to `0` for its runs and points `OC_HARNESS_BUDGET_FILE` at the run
+  directory, so a benchmark neither trips the guard nor charges the account's
+  real window. The suite's own cap is `--budget`.
+- **An empty plan reported success.** A plan with no tasks printed
+  "every task is done (0/0)" and exited 0. Nothing was done, so it now stops
+  with exit 2 and says the plan has no tasks.
+- **A blocked task did not stop the tasks after it.** The loop marked the
+  blocker and carried on, then reported it at the end. It now runs in file
+  order and stops at the first blocked task, so pending work before the
+  blocker still lands and work after it no longer builds on a failure.
+- **The task file was written back from the copy read at the start.** A
+  human edit or an agent deletion made mid-run was overwritten, and a deleted
+  task file was recreated. Write-back re-reads the file, refuses to write a
+  stale copy, and goes through `rename` so a reader never sees half a plan.
+- **A task was matched by title and position only.** With two same-titled
+  tasks, removing or renaming the first made the loop mark the second one
+  done. Matching now uses title + `change` + `files`, and an ambiguous match
+  stops the run instead of guessing.
+- **`verify: true` and `verify: echo ok` were treated as checks.** Both exit
+  0 whatever the agent did. A verify line made only of no-ops now blocks the
+  task before the agent runs; a no-op before a real command
+  (`true && npm test`) is still accepted.
+- **`doc` and `verify` opened their bash maps with `"*": "allow"`.** The
+  read-only lanes could write through `sed -i`, `awk` or `python3 -c`
+  regardless of their `edit: "deny"`. `doc` now defaults to deny and `verify`
+  defaults to ask, each with the commands its job actually needs.
+- **A run that hit the lane's step cap reported only "verify failed".** At the
+  cap OpenCode exits 0 with a text-only answer, so the loop saw a failed
+  check and blamed the change, while the log held the real reason. `run` now
+  counts `step_finish` events in the transcript and, when the count reaches
+  the role's configured cap, says so in the stop message and in the task's
+  recorded output, with the setting to raise.
+- **An agent that answered nothing was reported as a failed change.** A spent
+  model route shows up in one of two shapes. Some providers exit 0 with an
+  empty stdout and the reason on stderr: the loop then ran the verify against
+  an untouched tree and blocked the task for work the model never attempted.
+  Others hang until the ceiling kills them, which at the default budget is two
+  hours spent on one task with nothing to show for it.
+  `run` now stops with exit 3 when stdout is empty without running the verify,
+  and a timeout names the route and the limit instead of saying the agent
+  "failed". The task is marked blocked either way, with the stderr and the
+  advice in its output.
+- **The A/B ground through a quota error as if it were an answer.** The free
+  tier can print `{"type":"error"}` with "Free usage limit reached" on stdout
+  and exit 0; the suite read that as an empty stream and needed `--fail-fast`
+  runs before stopping. It now stops on the first one with exit 2 and the
+  provider's own words, because the refusal measures the provider and not the
+  kit.
+- **The kit-off arm made paid title calls.** `makeNokitHome` pinned `model`
+  but not `small_model`, so the title agent used opencode's default paid
+  model. A real run made 1,073 title calls to `gpt-5.4-nano`; every one was
+  refused for insufficient funds. Both are pinned now.
+- **A killed A/B left the agent running.** `spawnSync` blocks the event loop,
+  so Ctrl-C killed the parent and the child was reparented to systemd - one
+  survived 2h54m against the real auth store. The capture uses `spawn`, a
+  process group, a timeout timer that kills the group, and `SIGINT`/`SIGTERM`
+  handlers that do the same.
+- **`run` started without `opencode`, and outside a git work tree.** The
+  first was reported as a failed agent that blocked the first task; the
+  second silently switched off the `files:` scope check. Both are now
+  refused before the lock is taken, the second unless `--allow-extra-files`
+  is passed.
+- **The scope check counted the loop's own bookkeeping as an out-of-scope
+  edit.** The plugin writes `.space/telemetry.jsonl` on every agent call, so
+  a task that touched only the files it named was still blocked for changing
+  `.space/`. Loop state is excluded now; a real edit outside `files:` is
+  still caught.
+
+- **The A/B's wall-time row read the stream, not the ceiling.** A run killed
+  at 180s has a stream that stops at 58s, so the report printed it as a 58s
+  run and the wall column favored whichever arm was killed sooner. Each
+  sidecar now records the harness's own timer, `--rescore` prefers it, and a
+  resumed sidecar without one falls back to the stream's last event.
+
+### Added
+
+- **A per-session token budget, counted by the guard.** The free tier is one
+  shared window and opencode shows no number for it, so a long session spends
+  it silently - one made 599 requests and 12,025,980 input tokens, and the
+  first sign was the provider refusing the next request. The guard sums the
+  input tokens each `step-finish` part reports, warns once at
+  `OC_HARNESS_BUDGET_WARN` (default 2M) and refuses tool calls at
+  `OC_HARNESS_BUDGET_STOP` (default 5M) with the number and the fix. `0`
+  disables either. The count is per session, so a subagent is not charged to
+  the conversation that started it.
+- **`oc-harness use --no-skills`.** opencode writes every discovered skill's
+  name and description into the system message, so a large skill collection
+  is a fixed cost on every request. The flag turns the skill tool off for the
+  lanes, which carry their instructions in `prompts/`, and the config header
+  says so. Off by default.
+- **A shared-window ledger, so a new session does not reset the budget.** The
+  per-session count answered the wrong question: the free tier is one window
+  per account, and two sessions were measured spending one window 47 and 33
+  minutes apart. The guard now writes each session's spend to
+  `OC_HARNESS_BUDGET_FILE` (default `$XDG_STATE_HOME/opencode-harness-kit/budget.json`)
+  and sums the entries inside `OC_HARNESS_BUDGET_WINDOW_MINUTES` (default 60),
+  warning at `OC_HARNESS_BUDGET_WINDOW_WARN` (4M) and refusing tool calls at
+  `OC_HARNESS_BUDGET_WINDOW_STOP` (8M) in every session, including one opened
+  after the window ran out. The file is written atomically; an unreadable file
+  is moved to `budget.json.corrupt` and the window starts empty (if the move
+  cannot be written, the call fails closed), and entries older than six hours
+  are pruned.
+- **`eval/ab.mjs --out <dir>`**: one stream and one result per run, and a
+  rerun with the same `--out` skips the runs that already have a result. A
+  suite stopped by the quota or by a reboot resumes instead of starting over.
+  The directory also gets `run-manifest.json` with the suite hash, target
+  fingerprint and model; a resume whose manifest differs stops with exit 2
+  instead of mixing two experiments, and a directory from before the manifest
+  existed is adopted with a warning.
+- **`eval/ab.mjs --budget <tokens>`**: cap the suite's input tokens. Checked
+  before each run, so the run that crosses the cap is still measured; the next
+  one stops the suite with exit 2.
+- **A preflight run in `eval/ab.mjs`** (on by default, `--no-preflight` to
+  skip): one question to the model before the suite, exit 2 if it does not
+  answer, times out, or its stream carries an error event. Answering is the
+  test: exit 0 with zero steps and no text stops the suite too.
+- `oc-harness run --review`: after verify passes, ask the reviewer lane for
+  a schema'd verdict, `oc-harness/review@1`. A fail, prose, a malformed
+  object or a crashed reviewer stops the run with exit 5 and marks the task
+  blocked. The verdict is kept at `.space/review/<round>-<slug>.json`; the
+  contract is in `docs/REVIEW-VERDICT.md`.
+- `.space/run.lock`: one writer per task file. A lock whose pid is gone is
+  cleared; a live one is not. `--force-unlock` breaks one explicitly.
+- `run --allow-extra-files`: documented escape hatch for the scope check.
+- `eval/compare.mjs`: grade any agent CLI against the same suite with the same
+  citation resolver and mutation comparison. `eval/ab.mjs` can only compare
+  the kit with opencode, because it reads opencode's event stream; this runs a
+  plain command in a fresh copy of the subject, so an Aider or SWE-agent
+  number is comparable with the kit's own. A command that is not installed is
+  reported as unusable, not as a 0% pass rate.
+- `eval/ab.mjs --fail-fast <n>` (default 2): stop when `n` runs in a row produce
+  no tool call and no text. A spent free route hangs and times out rather than
+  failing, which looked like a benchmark that "stopped after a few minutes"
+  when it was really waiting out one dead run at a time. A timeout whose last
+  60 seconds (or a twentieth of the ceiling, whichever is smaller) are quiet
+  counts toward the streak too, because a spent route hangs before the
+  ceiling. The raw streams are kept for `--rescore`.
+
+### Changed
+
+- **The `build` lane's step cap is 120, not 40.** A session that reaches the
+  cap is forced into a text-only response, which in an interactive session
+  looks like the agent stopping mid-task after a few minutes and in
+  `oc-harness run` looks like a task that never finished. 40 steps is a
+  reasonable ceiling for one planned task and not for a session that is
+  building and testing a change. The other lanes keep their caps.
+- The README and `docs/EXIT-CODES.md` describe `run` as build + verify, not
+  plan → build → review → finish. It never planned, committed or pushed.
+- Exit code 5 is now used, and documented.
+
+---
+
+## [0.6.4] — 2026-10-23
+
+Fixed, each one found by re-reading what 0.6.3 published, not by running
+anything new:
+
+- **The install command in the README named a package that does not exist.**
+  The Install section said `npx oc-harness init` "from the registry", and
+  `oc-harness` is this kit's binary name, not a package anyone can fetch: the
+  command every new reader was pointed at returns `npm error 404 'oc-harness'
+  is not in this registry`. Every install path in the README now reads
+  `npx -y opencode-harness-kit@latest <cmd>`, bare `oc-harness` is defined
+  once as shorthand for one of the two real forms, and a test pins every
+  `npx` reference in the README to this package. Released because the npm
+  page is a snapshot: the README on npmjs.com renders from the published
+  tarball, so fixing it in git changes nothing for the next visitor.
+- **`profiles/index.json` listed `flint-40b-free` twice**, and the README's
+  shipped-profiles table repeated it with a second, differently-worded
+  note. `oc-harness list` printed twelve rows, `oc-harness init` offered
+  twelve choices, flint twice in both, and the README counted twelve
+  shipped profiles. Eleven profiles ship; one entry remains, carrying the
+  honest note of both. Tests now diff the index, the table and the loader,
+  with no Set to collapse the difference.
+- **`retention.yml` repeated `opencode/flint-40b-free`, and the last key
+  won.** The hand-rolled parser overwrites `models[id]` on every repeat, so
+  the vague 2026-10-22 stub silently replaced the 2026-10-23 record that
+  quotes the provider's terms verbatim - same retention class, wrong date,
+  lost detail. A test now scans the file for repeated keys.
+- **`install` from npx symlinked into the npx cache.** The installer always
+  symlinked the plugin, prompts, skills and GLOBAL.md back to the package
+  root - right from a git checkout ("edit here, live everywhere"), visible
+  breakage from an npx cache that `npm cache clean --force` deletes. From a
+  checkout nothing changes; from npx or any npm install, where `.git` is
+  absent, `install` now copies, so the harness on disk does not dangle the
+  day the cache is cleaned. `verify` and `uninstall` learn that a
+  byte-identical copy of the kit's file is the kit's, and a test installs
+  from a copy of the package with no `.git` into a temp HOME, asserts real
+  files where links used to go, and runs `verify` green.
+
+No runtime change: `dist/`, the plugin and every model-facing artifact are
+byte-for-byte what 0.6.3 shipped.
+
 ## [0.6.3] — 2026-10-23
 
 Nothing here changes the runtime. The plugin, the CLI and `dist/` are
@@ -210,15 +498,19 @@ default fleet was carrying a dead fallback rung in **all ten lanes** while
   release, and it is out of the repository now.
 
   A take is about one machine and one moment, and so is the recording of it.
-  The `.gitignore` already said as much: "a recording stays on the machine
-  that made it until somebody decides, deliberately, to publish it". The
-  decision went the other way, so the file went back where that rule keeps
-  it — `video/build/`, which is ignored — and the README no longer links a
-  video the repository does not carry.
-
-  What is published is still the old `docs/demo/pitch.mp4`: four minutes
-  forty-seven seconds, recorded live from a Mac terminal, with a voice. The
-  npm package never carried a video at all.
+  The `.gitignore` of that release said the same thing: "a recording stays on
+  the machine that made it until somebody decides, deliberately, to publish
+  it". The decision went the other way then, so the file went back where
+  that rule keeps it — `video/build/`, which is ignored — and the README no
+  longer links a video the repository does not carry. (Neither the rule nor
+  the recording survived 0.6.3, which removed the rule along with the
+  demonstration.)
+
+  At the time, what was published on GitHub was still the old
+  `docs/demo/pitch.mp4`: four minutes forty-seven seconds, recorded live
+  from a Mac terminal, with a voice. 0.6.3 removed it, and the history
+  rewrite of that release means no tag carries it today. The npm package
+  never carried a video at all.
 
 ## [0.5.8] — 2026-10-22
 
@@ -1868,7 +2160,8 @@ silent in the worst way:
 
 ---
 
-[Unreleased]: https://github.com/pham-canh/opencode-harness-kit/compare/v0.5.7...HEAD
+[0.7.0]: https://github.com/pham-canh/opencode-harness-kit/compare/v0.6.4...v0.7.0
+[Unreleased]: https://github.com/pham-canh/opencode-harness-kit/compare/v0.7.0...HEAD
 [0.5.7]: https://github.com/pham-canh/opencode-harness-kit/compare/v0.5.6...v0.5.7
 [0.5.6]: https://github.com/pham-canh/opencode-harness-kit/compare/v0.5.5...v0.5.6
 [0.5.5]: https://github.com/pham-canh/opencode-harness-kit/compare/v0.5.4...v0.5.5