opencode-harness-kit 0.6.3 → 0.7.0

package/README.md

@@ -20,7 +20,8 @@ git clone https://github.com/pham-canh/opencode-harness-kit.git
 ```
 
 The three steps `init` runs, spelled out for when you want to know what
-happened (prefix with `npx` or `./bin/oc-harness.mjs` to taste):
+happened (prefix each with `npx -y opencode-harness-kit@latest` or
+`./bin/oc-harness.mjs` to taste):
 
 ```bash
 oc-harness use fleet-safe    # compile the profile
@@ -48,7 +49,7 @@ lanes.json                   ← the jobs
 src/                         ← no model name anywhere in here
 ```
 
-Same plugin source, twelve shipped profiles, twelve different behaviours. The
+Same plugin source, eleven shipped profiles, eleven different behaviours. The
 test suite asserts that rather than taking it on faith.
 
 ### How a request travels
@@ -237,7 +238,6 @@ it. One file covers ten jobs either way.
 | `nemotron-3.5-lightning-free` | 5 | 262,144 | — | **may train** |
 | `mimo-v2.6-flash-free` | 5 | 200,000 | — | **may train** |
 | `big-pickle` | 5 | 160,000 | — | **may train** |
-| `flint-40b-free` | 5 | 262,144 | `max` | **may train** |
 
 The five-lane profiles run everything on one model, which is the right shape
 when you want one route and want to know exactly which one.
@@ -306,7 +306,7 @@ A CI job that failed on the second kind would go red on a Tuesday because
 somebody else changed a provider's limits, and a job that goes red for a
 reason you cannot act on is a job people stop reading.
 
-It is not a hypothetical. The day this was written, four of the twelve
+It is not a hypothetical. The day this was written, four of the shipped
 shipped profiles had already drifted from the live catalogue — the
 `temperature` flag had flipped on every one of them.
 
@@ -560,8 +560,10 @@ oc-harness auto       refresh + install + use + verify, asking nothing
 oc-harness auto --json   the same reconcile, as a report a script can read
 oc-harness use <id>   compile a profile into ~/.config/opencode/opencode.jsonc
 oc-harness use <id> --dryrun   print a summary of what would be written, write nothing
+oc-harness use <id> --no-skills   leave the skill list out of the lanes' prompt
 oc-harness refresh <id> re-derive a profile against the live model list
-oc-harness install    symlink the plugin, prompts, skills and GLOBAL.md
+oc-harness install    symlink the plugin, prompts, skills and GLOBAL.md into the config dir
+                      (from `npx`, they link into a package copy it makes first - see Install)
 oc-harness link       just the plugin, no prompts or skills
 oc-harness verify     check the install against the profile; non-zero on drift
 oc-harness agents     report agents the effort tiering does not reach
@@ -577,7 +579,7 @@ oc-harness uninstall remove the plugin, prompts, skills and GLOBAL.md again
 oc-harness cost      what the kit is doing on this project, in numbers
 oc-harness restore <file>  put one back
 oc-harness baseline   measure the current install
-oc-harness run        drive .space/tasks.md to done with the build agent
+oc-harness run        build and verify .space/tasks.md, one task at a time
 ```
 
 The measurement scripts are in `eval/`, and each one says what it measures
@@ -589,6 +591,7 @@ node eval/measure-prune.mjs     the pruner, on real sessions
 node eval/measure-install.mjs   before and after the install
 node eval/models.mjs            every model the profiles use, one line each
 node eval/ab.mjs                the A/B: the same task with and without the kit
+node eval/compare.mjs           the same suite against any agent CLI, same grader
 ```
 
 Every flag accepts both spellings: `--profile fleet-safe` and
@@ -857,12 +860,15 @@ One command if you are new here (`init` asks which free profile to run on,
 then runs the two commands below plus `verify`):
 
 ```bash
-npx oc-harness init
+npx -y opencode-harness-kit@latest init
 ```
 
-Two commands, in this order, if you already know which profile you want
-(`npx oc-harness …` from the registry, or `./bin/oc-harness.mjs …` from a
-checkout — they are the same program):
+Two commands, in this order, if you already know which profile you want.
+Two forms of the same program: `npx -y opencode-harness-kit@latest …` from
+the registry, or `./bin/oc-harness.mjs …` from a checkout. Bare `oc-harness`
+elsewhere in this README is shorthand for whichever form installed it —
+`npx` leaves no global binary, so every command already naming just
+`oc-harness` is one of the two above in disguise:
 
 ```bash
 oc-harness use fleet-safe   # writes opencode.jsonc
@@ -873,6 +879,15 @@ oc-harness install                # writes everything that file points at
 `GLOBAL.md` into `~/.config/opencode/`, so this repo stays the single source
 of truth and an edit here takes effect on the next opencode start.
 
+That is the checkout behaviour, and it relies on the repository staying put.
+Run through `npx` — or any package install — there is no repository to hold
+the truth, only a npm cache that `npm cache clean --force` can delete under
+it: the installer then COPIES the package once to
+`~/.config/opencode/kit/` and links into that instead, so `npm cache clean`
+cannot dangle the harness out from under its own config. Same commands, same
+config; `uninstall` takes the copy out with the links, and `verify` judges
+both modes by the same ownership rules.
+
 ### The one command that asks nothing
 
 `init` is the front door for a person: it asks which free profile to run on.
@@ -1020,6 +1035,20 @@ except the lanes it may hand work to. That is the mechanism behind "hand the
 job to the lane that has it": a lane that is not allowed to spawn anything
 cannot quietly become a general-purpose agent on the most expensive tier.
 
+Reaching the cap is not an error OpenCode reports: it exits 0 with a text-only
+answer. `oc-harness run` counts the turns in the transcript, so a verify that
+failed after the cap says "the agent used its whole 120-step allowance" instead
+of looking like an ordinary failure. The number quoted is the one in the
+installed config for the role being run.
+
+A spent model route is the other way a run looks like a failure and is not
+one. It shows up as an empty transcript - some providers exit 0 with it, others
+hang until the ceiling kills them. In the first shape `run` stops with exit 3
+and "the agent answered nothing" rather than verifying an untouched tree; in
+the second it names the route and the limit it waited out. Either way the
+advice is `opencode run --print-logs -m <model> hi`, and a smaller `--budget`
+fails sooner.
+
 ## The skill loop
 
 The lanes above are the *who*. The skills are the *order*, and they load by
@@ -1041,6 +1070,25 @@ the conversation instead of the requirement.
 Publishing is never a loop step. `space-finish` prepares the change; a human
 decides it ships.
 
+`oc-harness run` automates the build-and-verify half of that loop, not the
+whole loop. It hands one task at a time to an agent, re-runs that task's own
+`verify:` command, and writes the real output back into `.space/tasks.md`. It
+does not plan, commit or push: those stay with `space-plan`, `space-finish`
+and the human. A task with no `verify:` is refused rather than assumed done,
+and so is one whose whole verify line is a command that cannot fail (`true`,
+`echo ok`). A task that edits files outside its `files:` line stops the run
+unless `--allow-extra-files` says otherwise. Because that check reads
+`git status`, `run` itself needs a work tree; outside one it refuses to start
+rather than run unfenced. `--force-unlock` clears a lock held by a live pid
+that has been reused by something else.
+
+`--review` closes the review gap without pretending a sentence is a verdict:
+after verify passes, the `reviewer` lane is asked for one JSON object matching
+[`docs/REVIEW-VERDICT.md`](docs/REVIEW-VERDICT.md). A pass marks the task done.
+A fail, a malformed reply or a crashed reviewer marks it blocked and stops the
+run with exit 5. The validated verdict is written to `.space/review/`, so the
+decision can be audited after the fact.
+
 ## What it costs, measured
 
 The kit's own A/B - the same task with and without it, ten reps per arm, 120
@@ -1051,14 +1099,14 @@ so anyone can repeat it:
 |---|---|---|
 | **correct** | **54/60** | **55/60** |
 | tool calls | 12.5 | 8 |
-| input tokens | 56,205 | 10,540 |
+| uncached input | 56,205 | 10,540 |
 | final context | 39,928 | 12,718 |
 | wall seconds | 43 | 21 |
 | paths written outside the target | 0 | 0 |
 
 **On a task that fits in one read, the kit is a cost.** It spends over five
-times the input tokens and takes twice as long, and the answers are a tie -
-54 against 55, and a second run of the same 120 runs gave 55 against 54.
+times the uncached input tokens and takes twice as long, and the answers are a
+tie - 54 against 55, and a second run of the same 120 runs gave 55 against 54.
 The two edit tasks are a tie at 10/10, and the guard stopped no writes in
 either arm.
 
@@ -1100,6 +1148,97 @@ Whether it is worth it depends on whether the work is long. A repository of
 one-file lookups should not install this. The full table is in
 [eval/RESULTS.md](eval/RESULTS.md).
 
+## The token budget
+
+A free tier is one shared window: every request in every session draws on it,
+and when it runs out the provider answers with a limit error instead of
+tokens. opencode never shows a number for it. One real chat session made 599
+requests and 12,025,980 uncached input tokens; the first sign of trouble was
+the next request being refused, which reads as a broken model rather than a
+spent budget.
+
+The guard counts it. Each `step-finish` part reports what that request cost:
+
+| field | what it is | counted |
+|---|---|---|
+| `input` | the part of the prompt that was not in the provider's cache | yes |
+| `cache.read` | the cached prefix the model still had to read | yes |
+| `cache.write` | the prefix stored for the next request | yes |
+| `output`, `reasoning` | what the model wrote | no |
+
+The count is the whole context, not `input`. Counting `input` alone looks
+right and is not: with prompt caching, one measured stream reported 844, 1,289
+and 391 uncached tokens against 36,480, 7,808 and 9,088 cached ones. A budget
+built on the first column needed five to forty times the real spend to fire, and
+the window ran out first - which is exactly what happened. The guard sums the
+context per session, and at two thresholds it acts:
+
+| | default | what happens |
+|---|---|---|
+| `OC_HARNESS_BUDGET_WARN` | 2,000,000 | one toast: this session has used N context tokens |
+| `OC_HARNESS_BUDGET_STOP` | 5,000,000 | tool calls are refused, with the number and the fix |
+| `OC_HARNESS_TURN_STOP` | 300,000 | tool calls are refused after one request re-sends this much: the loop stop |
+
+```bash
+OC_HARNESS_BUDGET_STOP=2000000 OC_HARNESS_TURN_STOP=150000 opencode   # a smaller window for a long task
+OC_HARNESS_BUDGET_WARN=0 OC_HARNESS_BUDGET_STOP=0 OC_HARNESS_TURN_STOP=0 opencode   # off
+```
+
+The stop is a refusal with an explanation, not a crash: the model gets the
+message that the session hit its budget and that a new session is the fix. A
+subagent gets its own session and its own count, so its tokens are not
+charged to the conversation that started it.
+
+### The window is shared, so a new session does not reset it
+
+The per-session count is the wrong shape for the thing that actually runs out.
+The free tier is one window per account: session A spends it, and session B -
+opened a minute later to get a clean count - is refused on its first request.
+Two sessions were measured spending one window 47 and 33 minutes apart, so
+"open a new session" is not a fix once the window is gone.
+
+The guard therefore keeps a ledger, not just a counter. Every session's spend
+is written to one file and the window is the sum of the entries inside it:
+
+| | default | what happens |
+|---|---|---|
+| `OC_HARNESS_BUDGET_WINDOW_MINUTES` | 60 | how far back the sum reaches |
+| `OC_HARNESS_BUDGET_WINDOW_WARN` | 4,000,000 | one toast: the account has used N in the window |
+| `OC_HARNESS_BUDGET_WINDOW_STOP` | 8,000,000 | tool calls are refused in *every* session, old or new |
+| `OC_HARNESS_BUDGET_WINDOW_STICKY` | 1 | a crossed stop is recorded and holds until its entries age out (six hours) |
+
+The ledger is `OC_HARNESS_BUDGET_FILE`, default
+`$XDG_STATE_HOME/opencode-harness-kit/budget.json` (`~/.local/state/...` when
+`XDG_STATE_HOME` is unset).
+
+The file is written atomically (write, then rename); an unreadable file is
+moved to `budget.json.corrupt` and the window starts empty instead of failing
+the session or locking it for six hours. If the repair itself cannot be
+written - a read-only state directory, for example - the call fails closed
+rather than pretending the window is empty. Entries older than six hours are
+dropped on the next write. A crossed stop is written into the ledger as a
+marker, so a session opened after the one-hour sum rolls out is still refused.
+`OC_HARNESS_BUDGET_WINDOW_STICKY=0` turns that marker off. `0` disables either
+threshold. The stop is checked when the tool is called, not when the session
+starts, so a session opened after the window ran out is still refused - with
+the window's number, not the session's.
+
+A denial ends the turn: the generated config sets `experimental.continue_loop_on_deny: false`. With it on, opencode fed the guard's refusal back to the model as another turn. The measured session retried six times and made 138 requests / 7,559,016 context tokens in 29 minutes, each retry re-sending the whole prompt. `OC_HARNESS_TURN_STOP` is the second half: it blocks a single request that re-sends more than 300k, while the total still looks small.
+
+The other half is the fixed cost. opencode writes every discovered skill's
+name and description into the system message, so a machine with a large skill
+collection pays tens of thousands of tokens on every request, before the
+conversation starts. The lanes carry their instructions in `prompts/`, so:
+
+```bash
+oc-harness use fleet-safe --no-skills
+```
+
+omits the skill list from the lanes' prompt and says so in the config header.
+It is off by default: removing a capability is the user's call, not the kit's.
+
+It covers the ten lanes. **Subagents are a separate surface**: `--no-skills` does not change a subagent's `skill` tool. It does remove every entry in the generated agent map whose prompt path is `prompts/agents/<name>.txt` - the bundled ECC subagents - because their descriptions were loaded on every request while their prompts were read only when one was delegated to. A lane, or an agent whose prompt is any other path, is left alone; a hand-written agent that reuses the `prompts/agents/` path is removed like the bundle's own. The config header names every entry removed, and the `skill` tool on a subagent left in place is still your call, not the kit's.
+
 ## Known limits
 
 Read [docs/KNOWN-LIMITS.md](docs/KNOWN-LIMITS.md) before relying on failover.
@@ -1167,9 +1306,100 @@ wants a subject it can hand each run a private copy of:
 ```bash
 node eval/prepare.mjs --repo ~/work/lablab-harness --ref 8a674f5
 node eval/ab.mjs --target <printed path> --suite eval/tasks.json --reps 2 --sanitize
-node eval/ab.mjs --rescore <saved stream dir>   # re-grade without spawning agents
+node eval/ab.mjs --target <printed path> --suite eval/tasks.json \
+  --rescore <saved stream dir>   # re-grade without spawning agents
+node eval/premises.mjs --target <printed path> --suite eval/tasks.json
 ```
 
+`premises` is the freshness check for a suite: it resolves every `file:line`
+citation the tasks make against the target and exits non-zero on the first one
+that no longer exists. Run it after a subject moves, before spending runs or
+trusting a rescore. Point it at this repository and `eval/tasks-self.json` to
+check the kit's own suite.
+
+`--fail-fast <n>` (default 2) stops the suite when `n` runs in a row produce
+no tool call and no text, or when `n` timeouts in a row went quiet well before
+the per-run ceiling. That is what a spent free route looks like: the run hangs
+until the per-run ceiling and is recorded as a timeout, so without this a
+36-run suite spends hours measuring the provider's quota. A quota error the
+provider states outright (`Free usage limit reached`, an HTTP 429) stops the
+suite on the first run instead of waiting for two empty ones, and says so
+rather than scoring the refusal as a wrong answer. The raw streams are kept
+either way, so `--rescore <dir>` can still grade them later. The wall time
+comes from the sidecar, so a run killed at the ceiling is reported at the
+ceiling, not at the timestamp of its last event.
+
+Both arms pin `small_model` as well as `model`. `model` only covers the main
+loop; the title agent runs on `small_model`, whose default is a paid model, so
+an unpinned kit-off arm makes paid title calls for a benchmark that is meant
+to be free.
+
+The kit-off arm has its own `HOME` with no kit config in it, and that is
+enough to break it: opencode resolves `provider/model` against a catalogue it
+keeps in `~/.cache/opencode/models.json`, so an arm with a bare `HOME` reports
+`ProviderModelNotFoundError` for every model the live install knows. The
+catalogue is opencode's, not the kit's, so `eval/ab.mjs` symlinks the cache
+into the temporary home along with `~/.local/share/opencode`. If a run still
+fails before it starts, the error event is now fatal on the spot:
+
+- `--out <dir>` keeps each run's stream as `<id>-<arm>-<i>.json` next to a
+  `<id>-<arm>-<i>.result.json`, and a rerun with the same `--out` skips the
+  runs that already have a result. A sidecar with no verdict and no step is
+  retried instead of skipped: it is a run that never started, and skipping it
+  would make the hole permanent. A suite interrupted by the quota, or by a
+  reboot, continues instead of starting over. The directory gets a
+  `run-manifest.json`; resuming it against a different target, suite or model
+  is refused rather than mixed into one result.
+- `--budget <tokens>` caps the suite's context tokens - the same number the
+  guard counts, cached prefix included. It is checked before each run, so the
+  run that crosses the cap is still measured and counted; the next one stops
+  the suite with exit 2.
+- A preflight run (on by default, `--no-preflight` to skip) asks the model one
+  question before the suite and stops with exit 2 if it does not answer, if it
+  times out, if it exits cleanly with an empty stream, or if its stream carries
+  an error event. It costs one request and
+  saves the eighteen that a broken arm would otherwise spend being scored as
+  wrong answers. Its stream goes to a scratch directory, not `--out`, so
+  `--rescore` does not warn about a file it cannot grade.
+
+  It earns that cost on the day the window is out. `opencode run` then retries
+  the refused request three times and does not exit - one probe sat at zero
+  bytes until a 50-second timeout killed it, and the error text only appears in
+  `--print-logs`. A suite started in that state would have written 120
+  results, every one of them a timeout, and called it a benchmark. The
+  preflight turns it into `exit 2` and no result files. The 10-rep run that
+  produced the numbers in `eval/RESULTS.md` hit this at 47 of 120 runs: the 47
+  are real, the other 73 were never started, and resuming with the same
+  `--out` skips the 47. The window is also account-wide, not per model - a second free
+  model on the same provider answered "Free usage limit reached" too.
+
+The suite also turns the kit's own budget off for its runs and points the
+ledger at the run directory. A hundred and twenty runs spend more than the
+shared window's stop, and from that run on the guard would refuse tool calls -
+so the runs after it would be scored as wrong answers for a reason the suite
+is not measuring. The suite has its own cap, and the account's real window is
+not charged by a benchmark. The cap counts context, not uncached input: 47
+runs of one 10-rep attempt reported 1,284,594 uncached input tokens while the
+same streams carried 7,130,164 context tokens, so a cap set against the first
+number let the suite walk into the provider's wall.
+
+`eval/ab.mjs` knows how to read opencode's event stream, so it can only compare
+the kit with opencode. `eval/compare.mjs` grades any CLI the same way: it runs
+the command in a fresh copy of the subject, treats stdout as the answer, and
+uses the same citation resolver and mutation comparison. That is what makes a
+number for Aider or SWE-agent comparable with the kit's own.
+
+```bash
+node eval/compare.mjs --target <prepared path> --suite eval/tasks-self.json \
+  --reps 3 --timeout 300 \
+  --arm 'aider=aider --yes --no-auto-commits --message "$PROMPT"' \
+  --arm 'swe=python -m sweagent run --problem "$PROMPT"'
+```
+
+The command runs through `sh -c` with `$PROMPT`, `$TASK_ID`, `$DIR`, `$REP` and
+`$ARM` in the environment. A command that is not on PATH stops the run with
+exit 2 instead of being reported as a 0% pass rate.
+
 `npm test` and `npm run typecheck` need nothing but the repo, so CI runs those.
 The rest read `~/.local/share/opencode/opencode.db` and `~/.config/opencode`,
 which only exist on a machine that has run OpenCode with the kit installed —