opencode-harness-kit 0.6.4 → 0.7.0

package/CHANGELOG.md

@@ -12,6 +12,253 @@ is not happening.
 
 ---
 
+## [0.7.0] — 2026-10-25
+
+`oc-harness run` now stops for the reasons it already claimed to stop for,
+the read-only lanes no longer have a write-capable shell, and the token budget
+counts the context a request reads instead of only its uncached part.
+
+### Fixed
+
+- **A denied tool call was fed back as another turn, and the retry spent the
+  window.** `experimental.continue_loop_on_deny` was `true`, so opencode
+  handed the guard's refusal back to the model and continued the loop. The
+  measured session made 138 requests and 7,559,016 context tokens in 29
+  minutes on six `shared token window` denials, each retry re-sending the
+  whole prompt. The generated config now sets it to `false`, and the guard
+  blocks a single request that re-sends more than `OC_HARNESS_TURN_STOP`
+  (default 300k) with "do not retry".
+- **The in-event shared-window stop did not refuse anything.** When the
+  `step-finish` sum crossed `OC_HARNESS_BUDGET_WINDOW_STOP`, the guard set
+  `windowSpent` but never set `windowWhy`, so `sharedWindowSpent()` returned
+  an empty reason and every tool call was allowed. The reason is set where
+  the sum crosses, and the stop is written to the ledger with
+  `recordStop()` so a new session cannot outlive the provider's window.
+- **A corrupt ledger locked the machine for six hours, or vanished.** It is
+  now moved to `budget.json.corrupt` and the window starts empty; the broken
+  bytes are kept as evidence. `recordSpend()` and `recordStop()` repair
+  before writing instead of reading the corruption as an empty window.
+- **The bundled ECC subagents were loaded on every request.** Their
+  descriptions came from `prompts/agents/*.txt`, were not lanes, and so were
+  not covered by the stale-lane rule. They are removed from the generated
+  agent map when that is the prompt, and named in the config header; a
+  hand-written agent that reuses that prompt path is removed like the
+  bundle's own, while any other prompt is left alone.
+- **The token budget undercounted a cached session five to forty times, so it
+  never fired.** A `step-finish` part reports `input`, `cache.read`,
+  `cache.write`, `output` and `reasoning`. The guard summed `input` alone -
+  the part of the prompt that was not cached - while the cached prefix, which
+  is what a long conversation actually re-reads, was ignored. One measured
+  stream reported 844 uncached tokens against 36,480 cached ones on the same
+  step, then 391 against 9,088. The 2M warn and 5M stop were therefore five
+  to forty times the real spend, and the provider's window ran out before
+  either fired. `requestTokens()` in `src/budget.js` now counts `input + cache.read +
+  cache.write` for both the session total and the shared-window ledger; the
+  toasts and refusals say "context tokens" because that is what they count.
+- **The A/B's `--budget` had the same undercount.** The suite cap summed
+  `inputTokens`, so 120 runs reported about 1.3M of spend while their streams
+  carried 7.1M. `parseStream` now also returns `contextTokens`, the cap and the
+  report use it, and `--rescore` re-reads the old streams to put the corrected
+  number on them. `inputTokens` is still recorded so a table can be compared
+  with the ones written before this.
+- **The test suite wrote to the real window ledger.** `guard-hook.test.js`
+  exercised the guard with the default `OC_HARNESS_BUDGET_FILE`, so every test
+  run added its 1,500 and 600 tokens to
+  `~/.local/state/opencode-harness-kit/budget.json` - the live ledger showed
+  twelve test entries and nothing else. The tests now point the ledger at
+  their own temporary directory.
+- **`--rescore` graded a timeout as a wrong answer.** A run that hit the
+  ceiling has a stream that stops mid-sentence, which reads exactly like an
+  answer that cited nothing; only the sidecar knows which it was. Four runs
+  that timed out at 12 steps were re-scored as four wrong answers, and the
+  table then showed 18/23 where the honest count is 18/19.
+- **A resume crashed the report on a sidecar from an older version.** The
+  re-score reads `escaped`, `tools` and `text` from every result; a sidecar
+  written before those fields existed threw a `TypeError` after every run had
+  already been paid for. Missing fields are read as empty, and the corrected
+  token count is written back so a second resume does not depend on the stream
+  still being there.
+- **The A/B's preflight said "ok" to a run that had failed.** It checked only
+  the timeout and the quota, so a run that reported an error event and exited
+  in three seconds passed. The first suite with that preflight spent eighteen
+  runs learning that every kit-off arm had failed with
+  `ProviderModelNotFoundError`, scored as "no output, not scored". It now
+  reads the stream, and any error event stops the suite with exit 2 and the
+  provider's own message. An exit 0 with zero steps and no text is the same
+  shape - a spent route - and is refused the same way.
+- **The kit-off arm could not resolve the model it was told to use.** opencode
+  keeps its model catalogue in `~/.cache/opencode/models.json`, and the arm's
+  temporary `HOME` had no cache, so a bare `model:` string was a model the
+  installation had never heard of. The cache is opencode's, not the kit's, so
+  it is symlinked into the temporary home the same way `~/.local/share/opencode`
+  already was.
+- **An error in a measured run was counted as a wrong answer.** A run that
+  produced no tool call and no text, but did report an error, now stops the
+  suite at once with that error instead of waiting for `--fail-fast` and then
+  scoring the refusal.
+- **The lane counter's "no telemetry" test failed on a machine that had one
+  two directories up.** The lookup walks up to the project root by design, so
+  `/tmp/.space/telemetry.jsonl` - written by any `opencode run` started in
+  `/tmp` - made it find a file and fail for a reason that was not the code's.
+  The test now runs twelve levels down, past the walk-up's ceiling, so nothing
+  above its own tree is looked at; the walk-up itself is unchanged.
+- **The A/B was governed by the kit's own budget.** The kit-on arm runs on the
+  real config, so the guard's session and window thresholds applied to it: 120
+  runs spend more than the window's stop, and from the run that crossed it the
+  guard refused tool calls - every later run would be scored as a wrong answer
+  for a reason the suite is not measuring. The suite now sets all four
+  thresholds to `0` for its runs and points `OC_HARNESS_BUDGET_FILE` at the run
+  directory, so a benchmark neither trips the guard nor charges the account's
+  real window. The suite's own cap is `--budget`.
+- **An empty plan reported success.** A plan with no tasks printed
+  "every task is done (0/0)" and exited 0. Nothing was done, so it now stops
+  with exit 2 and says the plan has no tasks.
+- **A blocked task did not stop the tasks after it.** The loop marked the
+  blocker and carried on, then reported it at the end. It now runs in file
+  order and stops at the first blocked task, so pending work before the
+  blocker still lands and work after it no longer builds on a failure.
+- **The task file was written back from the copy read at the start.** A
+  human edit or an agent deletion made mid-run was overwritten, and a deleted
+  task file was recreated. Write-back re-reads the file, refuses to write a
+  stale copy, and goes through `rename` so a reader never sees half a plan.
+- **A task was matched by title and position only.** With two same-titled
+  tasks, removing or renaming the first made the loop mark the second one
+  done. Matching now uses title + `change` + `files`, and an ambiguous match
+  stops the run instead of guessing.
+- **`verify: true` and `verify: echo ok` were treated as checks.** Both exit
+  0 whatever the agent did. A verify line made only of no-ops now blocks the
+  task before the agent runs; a no-op before a real command
+  (`true && npm test`) is still accepted.
+- **`doc` and `verify` opened their bash maps with `"*": "allow"`.** The
+  read-only lanes could write through `sed -i`, `awk` or `python3 -c`
+  regardless of their `edit: "deny"`. `doc` now defaults to deny and `verify`
+  defaults to ask, each with the commands its job actually needs.
+- **A run that hit the lane's step cap reported only "verify failed".** At the
+  cap OpenCode exits 0 with a text-only answer, so the loop saw a failed
+  check and blamed the change, while the log held the real reason. `run` now
+  counts `step_finish` events in the transcript and, when the count reaches
+  the role's configured cap, says so in the stop message and in the task's
+  recorded output, with the setting to raise.
+- **An agent that answered nothing was reported as a failed change.** A spent
+  model route shows up in one of two shapes. Some providers exit 0 with an
+  empty stdout and the reason on stderr: the loop then ran the verify against
+  an untouched tree and blocked the task for work the model never attempted.
+  Others hang until the ceiling kills them, which at the default budget is two
+  hours spent on one task with nothing to show for it.
+  `run` now stops with exit 3 when stdout is empty without running the verify,
+  and a timeout names the route and the limit instead of saying the agent
+  "failed". The task is marked blocked either way, with the stderr and the
+  advice in its output.
+- **The A/B ground through a quota error as if it were an answer.** The free
+  tier can print `{"type":"error"}` with "Free usage limit reached" on stdout
+  and exit 0; the suite read that as an empty stream and needed `--fail-fast`
+  runs before stopping. It now stops on the first one with exit 2 and the
+  provider's own words, because the refusal measures the provider and not the
+  kit.
+- **The kit-off arm made paid title calls.** `makeNokitHome` pinned `model`
+  but not `small_model`, so the title agent used opencode's default paid
+  model. A real run made 1,073 title calls to `gpt-5.4-nano`; every one was
+  refused for insufficient funds. Both are pinned now.
+- **A killed A/B left the agent running.** `spawnSync` blocks the event loop,
+  so Ctrl-C killed the parent and the child was reparented to systemd - one
+  survived 2h54m against the real auth store. The capture uses `spawn`, a
+  process group, a timeout timer that kills the group, and `SIGINT`/`SIGTERM`
+  handlers that do the same.
+- **`run` started without `opencode`, and outside a git work tree.** The
+  first was reported as a failed agent that blocked the first task; the
+  second silently switched off the `files:` scope check. Both are now
+  refused before the lock is taken, the second unless `--allow-extra-files`
+  is passed.
+- **The scope check counted the loop's own bookkeeping as an out-of-scope
+  edit.** The plugin writes `.space/telemetry.jsonl` on every agent call, so
+  a task that touched only the files it named was still blocked for changing
+  `.space/`. Loop state is excluded now; a real edit outside `files:` is
+  still caught.
+
+- **The A/B's wall-time row read the stream, not the ceiling.** A run killed
+  at 180s has a stream that stops at 58s, so the report printed it as a 58s
+  run and the wall column favored whichever arm was killed sooner. Each
+  sidecar now records the harness's own timer, `--rescore` prefers it, and a
+  resumed sidecar without one falls back to the stream's last event.
+
+### Added
+
+- **A per-session token budget, counted by the guard.** The free tier is one
+  shared window and opencode shows no number for it, so a long session spends
+  it silently - one made 599 requests and 12,025,980 input tokens, and the
+  first sign was the provider refusing the next request. The guard sums the
+  input tokens each `step-finish` part reports, warns once at
+  `OC_HARNESS_BUDGET_WARN` (default 2M) and refuses tool calls at
+  `OC_HARNESS_BUDGET_STOP` (default 5M) with the number and the fix. `0`
+  disables either. The count is per session, so a subagent is not charged to
+  the conversation that started it.
+- **`oc-harness use --no-skills`.** opencode writes every discovered skill's
+  name and description into the system message, so a large skill collection
+  is a fixed cost on every request. The flag turns the skill tool off for the
+  lanes, which carry their instructions in `prompts/`, and the config header
+  says so. Off by default.
+- **A shared-window ledger, so a new session does not reset the budget.** The
+  per-session count answered the wrong question: the free tier is one window
+  per account, and two sessions were measured spending one window 47 and 33
+  minutes apart. The guard now writes each session's spend to
+  `OC_HARNESS_BUDGET_FILE` (default `$XDG_STATE_HOME/opencode-harness-kit/budget.json`)
+  and sums the entries inside `OC_HARNESS_BUDGET_WINDOW_MINUTES` (default 60),
+  warning at `OC_HARNESS_BUDGET_WINDOW_WARN` (4M) and refusing tool calls at
+  `OC_HARNESS_BUDGET_WINDOW_STOP` (8M) in every session, including one opened
+  after the window ran out. The file is written atomically; an unreadable file
+  is moved to `budget.json.corrupt` and the window starts empty (if the move
+  cannot be written, the call fails closed), and entries older than six hours
+  are pruned.
+- **`eval/ab.mjs --out <dir>`**: one stream and one result per run, and a
+  rerun with the same `--out` skips the runs that already have a result. A
+  suite stopped by the quota or by a reboot resumes instead of starting over.
+  The directory also gets `run-manifest.json` with the suite hash, target
+  fingerprint and model; a resume whose manifest differs stops with exit 2
+  instead of mixing two experiments, and a directory from before the manifest
+  existed is adopted with a warning.
+- **`eval/ab.mjs --budget <tokens>`**: cap the suite's input tokens. Checked
+  before each run, so the run that crosses the cap is still measured; the next
+  one stops the suite with exit 2.
+- **A preflight run in `eval/ab.mjs`** (on by default, `--no-preflight` to
+  skip): one question to the model before the suite, exit 2 if it does not
+  answer, times out, or its stream carries an error event. Answering is the
+  test: exit 0 with zero steps and no text stops the suite too.
+- `oc-harness run --review`: after verify passes, ask the reviewer lane for
+  a schema'd verdict, `oc-harness/review@1`. A fail, prose, a malformed
+  object or a crashed reviewer stops the run with exit 5 and marks the task
+  blocked. The verdict is kept at `.space/review/<round>-<slug>.json`; the
+  contract is in `docs/REVIEW-VERDICT.md`.
+- `.space/run.lock`: one writer per task file. A lock whose pid is gone is
+  cleared; a live one is not. `--force-unlock` breaks one explicitly.
+- `run --allow-extra-files`: documented escape hatch for the scope check.
+- `eval/compare.mjs`: grade any agent CLI against the same suite with the same
+  citation resolver and mutation comparison. `eval/ab.mjs` can only compare
+  the kit with opencode, because it reads opencode's event stream; this runs a
+  plain command in a fresh copy of the subject, so an Aider or SWE-agent
+  number is comparable with the kit's own. A command that is not installed is
+  reported as unusable, not as a 0% pass rate.
+- `eval/ab.mjs --fail-fast <n>` (default 2): stop when `n` runs in a row produce
+  no tool call and no text. A spent free route hangs and times out rather than
+  failing, which looked like a benchmark that "stopped after a few minutes"
+  when it was really waiting out one dead run at a time. A timeout whose last
+  60 seconds (or a twentieth of the ceiling, whichever is smaller) are quiet
+  counts toward the streak too, because a spent route hangs before the
+  ceiling. The raw streams are kept for `--rescore`.
+
+### Changed
+
+- **The `build` lane's step cap is 120, not 40.** A session that reaches the
+  cap is forced into a text-only response, which in an interactive session
+  looks like the agent stopping mid-task after a few minutes and in
+  `oc-harness run` looks like a task that never finished. 40 steps is a
+  reasonable ceiling for one planned task and not for a session that is
+  building and testing a change. The other lanes keep their caps.
+- The README and `docs/EXIT-CODES.md` describe `run` as build + verify, not
+  plan → build → review → finish. It never planned, committed or pushed.
+- Exit code 5 is now used, and documented.
+
+---
+
 ## [0.6.4] — 2026-10-23
 
 Fixed, each one found by re-reading what 0.6.3 published, not by running
@@ -1913,7 +2160,8 @@ silent in the worst way:
 
 ---
 
-[Unreleased]: https://github.com/pham-canh/opencode-harness-kit/compare/v0.5.7...HEAD
+[0.7.0]: https://github.com/pham-canh/opencode-harness-kit/compare/v0.6.4...v0.7.0
+[Unreleased]: https://github.com/pham-canh/opencode-harness-kit/compare/v0.7.0...HEAD
 [0.5.7]: https://github.com/pham-canh/opencode-harness-kit/compare/v0.5.6...v0.5.7
 [0.5.6]: https://github.com/pham-canh/opencode-harness-kit/compare/v0.5.5...v0.5.6
 [0.5.5]: https://github.com/pham-canh/opencode-harness-kit/compare/v0.5.4...v0.5.5

package/README.md

@@ -560,6 +560,7 @@ oc-harness auto       refresh + install + use + verify, asking nothing
 oc-harness auto --json   the same reconcile, as a report a script can read
 oc-harness use <id>   compile a profile into ~/.config/opencode/opencode.jsonc
 oc-harness use <id> --dryrun   print a summary of what would be written, write nothing
+oc-harness use <id> --no-skills   leave the skill list out of the lanes' prompt
 oc-harness refresh <id> re-derive a profile against the live model list
 oc-harness install    symlink the plugin, prompts, skills and GLOBAL.md into the config dir
                       (from `npx`, they link into a package copy it makes first - see Install)
@@ -578,7 +579,7 @@ oc-harness uninstall remove the plugin, prompts, skills and GLOBAL.md again
 oc-harness cost      what the kit is doing on this project, in numbers
 oc-harness restore <file>  put one back
 oc-harness baseline   measure the current install
-oc-harness run        drive .space/tasks.md to done with the build agent
+oc-harness run        build and verify .space/tasks.md, one task at a time
 ```
 
 The measurement scripts are in `eval/`, and each one says what it measures
@@ -590,6 +591,7 @@ node eval/measure-prune.mjs     the pruner, on real sessions
 node eval/measure-install.mjs   before and after the install
 node eval/models.mjs            every model the profiles use, one line each
 node eval/ab.mjs                the A/B: the same task with and without the kit
+node eval/compare.mjs           the same suite against any agent CLI, same grader
 ```
 
 Every flag accepts both spellings: `--profile fleet-safe` and
@@ -1033,6 +1035,20 @@ except the lanes it may hand work to. That is the mechanism behind "hand the
 job to the lane that has it": a lane that is not allowed to spawn anything
 cannot quietly become a general-purpose agent on the most expensive tier.
 
+Reaching the cap is not an error OpenCode reports: it exits 0 with a text-only
+answer. `oc-harness run` counts the turns in the transcript, so a verify that
+failed after the cap says "the agent used its whole 120-step allowance" instead
+of looking like an ordinary failure. The number quoted is the one in the
+installed config for the role being run.
+
+A spent model route is the other way a run looks like a failure and is not
+one. It shows up as an empty transcript - some providers exit 0 with it, others
+hang until the ceiling kills them. In the first shape `run` stops with exit 3
+and "the agent answered nothing" rather than verifying an untouched tree; in
+the second it names the route and the limit it waited out. Either way the
+advice is `opencode run --print-logs -m <model> hi`, and a smaller `--budget`
+fails sooner.
+
 ## The skill loop
 
 The lanes above are the *who*. The skills are the *order*, and they load by
@@ -1054,6 +1070,25 @@ the conversation instead of the requirement.
 Publishing is never a loop step. `space-finish` prepares the change; a human
 decides it ships.
 
+`oc-harness run` automates the build-and-verify half of that loop, not the
+whole loop. It hands one task at a time to an agent, re-runs that task's own
+`verify:` command, and writes the real output back into `.space/tasks.md`. It
+does not plan, commit or push: those stay with `space-plan`, `space-finish`
+and the human. A task with no `verify:` is refused rather than assumed done,
+and so is one whose whole verify line is a command that cannot fail (`true`,
+`echo ok`). A task that edits files outside its `files:` line stops the run
+unless `--allow-extra-files` says otherwise. Because that check reads
+`git status`, `run` itself needs a work tree; outside one it refuses to start
+rather than run unfenced. `--force-unlock` clears a lock held by a live pid
+that has been reused by something else.
+
+`--review` closes the review gap without pretending a sentence is a verdict:
+after verify passes, the `reviewer` lane is asked for one JSON object matching
+[`docs/REVIEW-VERDICT.md`](docs/REVIEW-VERDICT.md). A pass marks the task done.
+A fail, a malformed reply or a crashed reviewer marks it blocked and stops the
+run with exit 5. The validated verdict is written to `.space/review/`, so the
+decision can be audited after the fact.
+
 ## What it costs, measured
 
 The kit's own A/B - the same task with and without it, ten reps per arm, 120
@@ -1064,14 +1099,14 @@ so anyone can repeat it:
 |---|---|---|
 | **correct** | **54/60** | **55/60** |
 | tool calls | 12.5 | 8 |
-| input tokens | 56,205 | 10,540 |
+| uncached input | 56,205 | 10,540 |
 | final context | 39,928 | 12,718 |
 | wall seconds | 43 | 21 |
 | paths written outside the target | 0 | 0 |
 
 **On a task that fits in one read, the kit is a cost.** It spends over five
-times the input tokens and takes twice as long, and the answers are a tie -
-54 against 55, and a second run of the same 120 runs gave 55 against 54.
+times the uncached input tokens and takes twice as long, and the answers are a
+tie - 54 against 55, and a second run of the same 120 runs gave 55 against 54.
 The two edit tasks are a tie at 10/10, and the guard stopped no writes in
 either arm.
 
@@ -1113,6 +1148,97 @@ Whether it is worth it depends on whether the work is long. A repository of
 one-file lookups should not install this. The full table is in
 [eval/RESULTS.md](eval/RESULTS.md).
 
+## The token budget
+
+A free tier is one shared window: every request in every session draws on it,
+and when it runs out the provider answers with a limit error instead of
+tokens. opencode never shows a number for it. One real chat session made 599
+requests and 12,025,980 uncached input tokens; the first sign of trouble was
+the next request being refused, which reads as a broken model rather than a
+spent budget.
+
+The guard counts it. Each `step-finish` part reports what that request cost:
+
+| field | what it is | counted |
+|---|---|---|
+| `input` | the part of the prompt that was not in the provider's cache | yes |
+| `cache.read` | the cached prefix the model still had to read | yes |
+| `cache.write` | the prefix stored for the next request | yes |
+| `output`, `reasoning` | what the model wrote | no |
+
+The count is the whole context, not `input`. Counting `input` alone looks
+right and is not: with prompt caching, one measured stream reported 844, 1,289
+and 391 uncached tokens against 36,480, 7,808 and 9,088 cached ones. A budget
+built on the first column needed five to forty times the real spend to fire, and
+the window ran out first - which is exactly what happened. The guard sums the
+context per session, and at two thresholds it acts:
+
+| | default | what happens |
+|---|---|---|
+| `OC_HARNESS_BUDGET_WARN` | 2,000,000 | one toast: this session has used N context tokens |
+| `OC_HARNESS_BUDGET_STOP` | 5,000,000 | tool calls are refused, with the number and the fix |
+| `OC_HARNESS_TURN_STOP` | 300,000 | tool calls are refused after one request re-sends this much: the loop stop |
+
+```bash
+OC_HARNESS_BUDGET_STOP=2000000 OC_HARNESS_TURN_STOP=150000 opencode   # a smaller window for a long task
+OC_HARNESS_BUDGET_WARN=0 OC_HARNESS_BUDGET_STOP=0 OC_HARNESS_TURN_STOP=0 opencode   # off
+```
+
+The stop is a refusal with an explanation, not a crash: the model gets the
+message that the session hit its budget and that a new session is the fix. A
+subagent gets its own session and its own count, so its tokens are not
+charged to the conversation that started it.
+
+### The window is shared, so a new session does not reset it
+
+The per-session count is the wrong shape for the thing that actually runs out.
+The free tier is one window per account: session A spends it, and session B -
+opened a minute later to get a clean count - is refused on its first request.
+Two sessions were measured spending one window 47 and 33 minutes apart, so
+"open a new session" is not a fix once the window is gone.
+
+The guard therefore keeps a ledger, not just a counter. Every session's spend
+is written to one file and the window is the sum of the entries inside it:
+
+| | default | what happens |
+|---|---|---|
+| `OC_HARNESS_BUDGET_WINDOW_MINUTES` | 60 | how far back the sum reaches |
+| `OC_HARNESS_BUDGET_WINDOW_WARN` | 4,000,000 | one toast: the account has used N in the window |
+| `OC_HARNESS_BUDGET_WINDOW_STOP` | 8,000,000 | tool calls are refused in *every* session, old or new |
+| `OC_HARNESS_BUDGET_WINDOW_STICKY` | 1 | a crossed stop is recorded and holds until its entries age out (six hours) |
+
+The ledger is `OC_HARNESS_BUDGET_FILE`, default
+`$XDG_STATE_HOME/opencode-harness-kit/budget.json` (`~/.local/state/...` when
+`XDG_STATE_HOME` is unset).
+
+The file is written atomically (write, then rename); an unreadable file is
+moved to `budget.json.corrupt` and the window starts empty instead of failing
+the session or locking it for six hours. If the repair itself cannot be
+written - a read-only state directory, for example - the call fails closed
+rather than pretending the window is empty. Entries older than six hours are
+dropped on the next write. A crossed stop is written into the ledger as a
+marker, so a session opened after the one-hour sum rolls out is still refused.
+`OC_HARNESS_BUDGET_WINDOW_STICKY=0` turns that marker off. `0` disables either
+threshold. The stop is checked when the tool is called, not when the session
+starts, so a session opened after the window ran out is still refused - with
+the window's number, not the session's.
+
+A denial ends the turn: the generated config sets `experimental.continue_loop_on_deny: false`. With it on, opencode fed the guard's refusal back to the model as another turn. The measured session retried six times and made 138 requests / 7,559,016 context tokens in 29 minutes, each retry re-sending the whole prompt. `OC_HARNESS_TURN_STOP` is the second half: it blocks a single request that re-sends more than 300k, while the total still looks small.
+
+The other half is the fixed cost. opencode writes every discovered skill's
+name and description into the system message, so a machine with a large skill
+collection pays tens of thousands of tokens on every request, before the
+conversation starts. The lanes carry their instructions in `prompts/`, so:
+
+```bash
+oc-harness use fleet-safe --no-skills
+```
+
+omits the skill list from the lanes' prompt and says so in the config header.
+It is off by default: removing a capability is the user's call, not the kit's.
+
+It covers the ten lanes. **Subagents are a separate surface**: `--no-skills` does not change a subagent's `skill` tool. It does remove every entry in the generated agent map whose prompt path is `prompts/agents/<name>.txt` - the bundled ECC subagents - because their descriptions were loaded on every request while their prompts were read only when one was delegated to. A lane, or an agent whose prompt is any other path, is left alone; a hand-written agent that reuses the `prompts/agents/` path is removed like the bundle's own. The config header names every entry removed, and the `skill` tool on a subagent left in place is still your call, not the kit's.
+
 ## Known limits
 
 Read [docs/KNOWN-LIMITS.md](docs/KNOWN-LIMITS.md) before relying on failover.
@@ -1180,9 +1306,100 @@ wants a subject it can hand each run a private copy of:
 ```bash
 node eval/prepare.mjs --repo ~/work/lablab-harness --ref 8a674f5
 node eval/ab.mjs --target <printed path> --suite eval/tasks.json --reps 2 --sanitize
-node eval/ab.mjs --rescore <saved stream dir>   # re-grade without spawning agents
+node eval/ab.mjs --target <printed path> --suite eval/tasks.json \
+  --rescore <saved stream dir>   # re-grade without spawning agents
+node eval/premises.mjs --target <printed path> --suite eval/tasks.json
 ```
 
+`premises` is the freshness check for a suite: it resolves every `file:line`
+citation the tasks make against the target and exits non-zero on the first one
+that no longer exists. Run it after a subject moves, before spending runs or
+trusting a rescore. Point it at this repository and `eval/tasks-self.json` to
+check the kit's own suite.
+
+`--fail-fast <n>` (default 2) stops the suite when `n` runs in a row produce
+no tool call and no text, or when `n` timeouts in a row went quiet well before
+the per-run ceiling. That is what a spent free route looks like: the run hangs
+until the per-run ceiling and is recorded as a timeout, so without this a
+36-run suite spends hours measuring the provider's quota. A quota error the
+provider states outright (`Free usage limit reached`, an HTTP 429) stops the
+suite on the first run instead of waiting for two empty ones, and says so
+rather than scoring the refusal as a wrong answer. The raw streams are kept
+either way, so `--rescore <dir>` can still grade them later. The wall time
+comes from the sidecar, so a run killed at the ceiling is reported at the
+ceiling, not at the timestamp of its last event.
+
+Both arms pin `small_model` as well as `model`. `model` only covers the main
+loop; the title agent runs on `small_model`, whose default is a paid model, so
+an unpinned kit-off arm makes paid title calls for a benchmark that is meant
+to be free.
+
+The kit-off arm has its own `HOME` with no kit config in it, and that is
+enough to break it: opencode resolves `provider/model` against a catalogue it
+keeps in `~/.cache/opencode/models.json`, so an arm with a bare `HOME` reports
+`ProviderModelNotFoundError` for every model the live install knows. The
+catalogue is opencode's, not the kit's, so `eval/ab.mjs` symlinks the cache
+into the temporary home along with `~/.local/share/opencode`. If a run still
+fails before it starts, the error event is now fatal on the spot:
+
+- `--out <dir>` keeps each run's stream as `<id>-<arm>-<i>.json` next to a
+  `<id>-<arm>-<i>.result.json`, and a rerun with the same `--out` skips the
+  runs that already have a result. A sidecar with no verdict and no step is
+  retried instead of skipped: it is a run that never started, and skipping it
+  would make the hole permanent. A suite interrupted by the quota, or by a
+  reboot, continues instead of starting over. The directory gets a
+  `run-manifest.json`; resuming it against a different target, suite or model
+  is refused rather than mixed into one result.
+- `--budget <tokens>` caps the suite's context tokens - the same number the
+  guard counts, cached prefix included. It is checked before each run, so the
+  run that crosses the cap is still measured and counted; the next one stops
+  the suite with exit 2.
+- A preflight run (on by default, `--no-preflight` to skip) asks the model one
+  question before the suite and stops with exit 2 if it does not answer, if it
+  times out, if it exits cleanly with an empty stream, or if its stream carries
+  an error event. It costs one request and
+  saves the eighteen that a broken arm would otherwise spend being scored as
+  wrong answers. Its stream goes to a scratch directory, not `--out`, so
+  `--rescore` does not warn about a file it cannot grade.
+
+  It earns that cost on the day the window is out. `opencode run` then retries
+  the refused request three times and does not exit - one probe sat at zero
+  bytes until a 50-second timeout killed it, and the error text only appears in
+  `--print-logs`. A suite started in that state would have written 120
+  results, every one of them a timeout, and called it a benchmark. The
+  preflight turns it into `exit 2` and no result files. The 10-rep run that
+  produced the numbers in `eval/RESULTS.md` hit this at 47 of 120 runs: the 47
+  are real, the other 73 were never started, and resuming with the same
+  `--out` skips the 47. The window is also account-wide, not per model - a second free
+  model on the same provider answered "Free usage limit reached" too.
+
+The suite also turns the kit's own budget off for its runs and points the
+ledger at the run directory. A hundred and twenty runs spend more than the
+shared window's stop, and from that run on the guard would refuse tool calls -
+so the runs after it would be scored as wrong answers for a reason the suite
+is not measuring. The suite has its own cap, and the account's real window is
+not charged by a benchmark. The cap counts context, not uncached input: 47
+runs of one 10-rep attempt reported 1,284,594 uncached input tokens while the
+same streams carried 7,130,164 context tokens, so a cap set against the first
+number let the suite walk into the provider's wall.
+
+`eval/ab.mjs` knows how to read opencode's event stream, so it can only compare
+the kit with opencode. `eval/compare.mjs` grades any CLI the same way: it runs
+the command in a fresh copy of the subject, treats stdout as the answer, and
+uses the same citation resolver and mutation comparison. That is what makes a
+number for Aider or SWE-agent comparable with the kit's own.
+
+```bash
+node eval/compare.mjs --target <prepared path> --suite eval/tasks-self.json \
+  --reps 3 --timeout 300 \
+  --arm 'aider=aider --yes --no-auto-commits --message "$PROMPT"' \
+  --arm 'swe=python -m sweagent run --problem "$PROMPT"'
+```
+
+The command runs through `sh -c` with `$PROMPT`, `$TASK_ID`, `$DIR`, `$REP` and
+`$ARM` in the environment. A command that is not on PATH stops the run with
+exit 2 instead of being reported as a 0% pass rate.
+
 `npm test` and `npm run typecheck` need nothing but the repo, so CI runs those.
 The rest read `~/.local/share/opencode/opencode.db` and `~/.config/opencode`,
 which only exist on a machine that has run OpenCode with the kit installed —