opencode-harness-kit 0.4.4 → 0.4.6

package/CHANGELOG.md

@@ -12,6 +12,57 @@ is not happening.
 
 ---
 
+## [0.4.6] — 2026-10-21
+
+Published. The tag is what cut it: `git tag v0.4.6 && git push origin main
+v0.4.6`, and the publish job watches for exactly that.
+
+### Fixed
+
+- **The model measurement table said `0 null`** when a model that publishes
+  a `low` effort level failed every request for that variant. A row of
+  zeroes reads like a measurement of nothing; a row that says the request
+  failed says what happened.
+
+## [0.4.5] — 2026-10-21
+
+Published. The tag is what cut it: `git tag v0.4.5 && git push origin main
+v0.4.5`, and the publish job watches for exactly that.
+
+This release is the measurement side of the kit: the numbers in the routing
+table were taken once, by hand, and could not be taken again. Now they can
+be, and the page that carries them says which measurement it is holding.
+
+### Added
+
+- **`eval/models.mjs`** - measures every model the shipped profiles use, one
+  line each, the same prompt for all of them, with the `low` effort variant
+  as well. It is what the routing table in `docs/ROUTING.md` is measured
+  with, and it did not exist: the numbers in that table had been taken once,
+  by hand, and could not be taken again.
+
+### Changed
+
+- **`docs/ROUTING.md` is measured again**, on 2026-10-21, and the two sets
+  of numbers are now labelled for what they measure. The install tables are
+  the telemetry of a working session; the new one is a direct call, from the
+  process start, which is the wait a lane would actually cost. Beam is still
+  the fastest of the ten, but by under a second over the next three, so the
+  reason to pick it is the 1M context as much as the speed. GLM is level
+  with Nemotron 3 Ultra for review, so the safe ceiling costs nothing
+  there. LongCat is twice Beam on the same one-line question.
+
+- **The 0.4.4 A/B table above was corrected.** It was written from the run
+  the grader mis-scored, and it said the kit was worse. The shipped
+  measurement is a tie.
+
+### Fixed
+
+- **`eval/models.mjs` accepted a mistyped `--reps`** and measured nothing
+  while reporting success - the same class of bug this kit keeps finding in
+  other places. It also ignored `--reps=5`, did not say why a model failed,
+  and left a scratch directory behind when it was killed.
+
 ## [0.4.4] — 2026-10-21
 
 Published. The tag is what cut it: `git tag v0.4.4 && git push origin main
@@ -115,15 +166,6 @@ The field in the log was renamed from `savedBytes` to `savedChars`, and
 the log is project-local and persistent: every project that upgraded had
 the old records and read them as zero. The old name is still read.
 
-**The published type declarations did not resolve for a consumer.**
-
-They import a type from `@opencode-ai/plugin`, which was a devDependency, so
-a consumer that installed the package type-checked against a type it did
-not have. The subpath exports - `opencode-harness-kit/profile` and the rest
-- declared no types at all, even though the declarations were built and
-shipped. The plugin is now an optional peer dependency and every subpath
-declares its types.
-
 **`oc-harness cost` reported characters as bytes.**
 
 The pruner counts characters, and the number it printed said "bytes", which
@@ -301,9 +343,11 @@ improvement from a NULL average.
   parts UNPRUNED - the pruner rewrites the outgoing request, not the history
   - so the stored parts of a real session are exactly what the pruner was
   given, and replaying the real pruner over them measures exactly what it
-  removed, on the same messages with and without. On this install: **80.5% of
-  the context across 60 real sessions**, 6,608 parts stubbed - the number
-  `npm run measure:prune` prints.
+  removed, on the same messages with and without. On this install, measured on
+  2026-10-20: **80.5% of the context across 60 real sessions**, 6,608 parts
+  stubbed. The number moves with the sessions, so it is a measurement of one
+  install at one moment - run `npm run measure:prune` for the number on
+  yours.
 
   The measurement scripts also run without the sqlite3 CLI now. python3's
   built-in sqlite3 module does the same job, and python3 is on more machines
@@ -469,23 +513,33 @@ improvement from a NULL average.
   check is not a result. The new suite runs against the kit's own tree at a
   pinned commit, so `git clone && git checkout <sha>` reproduces it exactly.
 
-  Ten reps per arm, 120 agent runs, and the result is not flattering:
+  Ten reps per arm, 120 agent runs, on the same model in both arms:
 
   | metric (median) | kit on | kit off |
   |---|---|---|
-  | **correct** | **52/60** | **59/59** |
-  | tool calls | 11.5 | 3 |
-  | input tokens | 51,449 | 4,551 |
-  | final context | 39,603 | 10,855 |
-  | wall seconds | 43.6 | 32.5 |
+  | **correct** | **54/60** | **55/60** |
+  | tool calls | 12.5 | 8 |
+  | input tokens | 56,205 | 10,540 |
+  | final context | 39,928 | 12,718 |
+  | wall seconds | 43 | 21 |
   | paths written outside the target | 0 | 0 |
 
   **On a task that fits in one read, the kit is a cost.** Its instructions
-  cost eleven times the input tokens and four times the tool calls, the run
-  takes a third longer, and the answers are worse. The rule that says *locate
-  before you read* is a waste on a one-file question, and the tool counts show
-  it. The two edit tasks are a tie at 10/10, and the guard stopped no writes
-  in either arm.
+  cost over five times the input tokens and half again as many tool calls,
+  and the run takes twice as long. The answers are a tie - 54 against 55,
+  and a second run of the same 120 runs gave 55 against 54, the other way.
+  The rule that says *locate before you read* is a waste on a one-file
+  question, and the tool counts show it. The two edit tasks are a tie at
+  10/10, and the guard stopped no writes in either arm.
+
+  **The first version of this table said the kit was worse - 52/60 against
+  59/59 - and the grader was why.** It could not read a citation written as
+  `**File:** x` on one line and `**Line:** 105` on the next, which is the
+  shape the kit's own instructions produce, while the kit-off arm answered
+  in prose and was graded by a pattern that handled prose. Re-scoring the
+  same 120 answers with the fixed grader puts the kit ahead on the lookup
+  tasks. A harness that measures the harness is worse than one that measures
+  nothing.
 
   The value the kit does have is in long sessions - the pruner removes 80.5%
   of the context across real ones - and in process, which a grader that

package/CONTRIBUTING.md

@@ -69,6 +69,13 @@ config, run the layers individually and say so in the PR.
 not part of any check, and a PR does not need it. `npm run measure` does not
 spawn an agent either — it queries the install database read-only.
 
+`eval/models.mjs` spawns real agents too, one line each, against the live
+tier: it is what the routing table in `docs/ROUTING.md` is measured with,
+and it costs a request per model per rep. It is not part of any check
+either. If you change a lane, a model or a threshold, run it and put the
+new numbers in the same commit — a routing table with numbers from a
+different day is a table nobody can check.
+
 ## House rules
 
 - Node 22 or newer. The CLI uses `node:` prefixed builtins, so an older Node

package/README.md

@@ -80,16 +80,22 @@ about your prompts.
 
 | lane | job | model | measured |
 |---|---|---|---|
-| `build` | write and verify the change | MiniMax M3.1 Flash | 3.1 s |
-| `plan` | turn a request into a plan | Beam | **2.6 s** |
-| `reviewer` | read a change in two passes | Nemotron 3 Ultra | 4.7 s (34 s cold) |
-| `locate` | find where something lives | Nemotron 3.5 Lightning | 3.7 s |
-| `verify` | run the project's checks | Big Pickle | 6.5 s |
-| `vision` | read an image or a screenshot | MiMo-V2.6 Flash | 19.2 s |
-| `doc` | read a long document or a spec | GLM 5.3 Flash | 5.4 s |
-| `schema` | produce JSON or a JSON schema | DeepSeek V4.1 Flash | 3.4 s |
-| `sweep` | sweep a lot of code mechanically | Nemotron 3.5 Super VL | 4.2 s |
-| `longform` | read a megabyte, or write alternatives | LongCat 2.5 Preview | 9.4 s |
+| `build` | write and verify the change | MiniMax M3.1 Flash | 6.9 s |
+| `plan` | turn a request into a plan | Beam | **6.1 s** |
+| `reviewer` | read a change in two passes | Nemotron 3 Ultra | 7.8 s (34 s cold) |
+| `locate` | find where something lives | Nemotron 3.5 Lightning | 6.9 s |
+| `verify` | run the project's checks | Big Pickle | 10.4 s |
+| `vision` | read an image or a screenshot | MiMo-V2.6 Flash | 9.3 s (19 s with an image) |
+| `doc` | read a long document or a spec | GLM 5.3 Flash | 7.4 s |
+| `schema` | produce JSON or a JSON schema | DeepSeek V4.1 Flash | 10.7 s |
+| `sweep` | sweep a lot of code mechanically | Nemotron 3.5 Super VL | 8.4 s |
+| `longform` | read a megabyte, or write alternatives | LongCat 2.5 Preview | 12.0 s |
+
+The `measured` column is the median of five one-line requests to that model
+on 2026-10-21 - the same prompt for every model, from the process start, so
+it is the wait a lane would cost.
+[`docs/ROUTING.md`](docs/ROUTING.md) has the method, the install telemetry
+beside it, and the command that produced it.
 
 Each lane is an OpenCode agent with its own model, its own context window and
 its own effort level. You do not switch models: you hand the job to the lane
@@ -103,11 +109,12 @@ agent=build    modelID=minimax-m3.1-flash-free
 ```
 
 Why ten lanes and not one model: the free tier is genuinely uneven, and the
-difference is measurable. Beam answers in 2.6 seconds with a 1M window;
-MiMo reads an image in 19 and MiniMax reads the same image in 8; Nemotron
-Super VL is the only model on the tier that publishes a `none` effort level,
-which makes "reasoning off" a job of its own. Putting each on the job it is
-good at is worth more than any single "best" model.
+difference is measurable. Beam answers in 6.1 seconds with a 1M window, and
+the four fastest models are inside a second of each other; MiMo reads an
+image in 19 and MiniMax reads the same image in 8; Nemotron Super VL is the
+only model on the tier that publishes a `none` effort level, which makes
+"reasoning off" a job of its own. Putting each on the job it is good at is
+worth more than any single "best" model.
 
 The full table, with the measurements behind every choice and the three
 capabilities the tier turns out not to have, is in
@@ -324,15 +331,16 @@ tier cannot do.
 | Node | **22 or newer** — `engines` in package.json, and the CLI uses `node:` prefixed builtins. Node 20 fails at import time, not with a friendly message. |
 | npm | 10 or newer, for `npm ci` when installing from a checkout |
 | OS | macOS, Linux and Windows. The tests run on `ubuntu-latest`. |
-| Shell | none needed to *use* the kit — no `bash`, `zsh` or `fish` setup. Two commands shell out to external binaries: `doctor` and `refresh` call `curl` to read models.dev, and `baseline` and the measurement scripts call `sqlite3`. Everything else, including `install`, `use` and `verify`, works offline. |
+| Shell | none needed to *use* the kit — no `bash`, `zsh` or `fish` setup. `doctor` and `refresh` call `curl` to read models.dev. `baseline` and the measurement scripts read the install database with `sqlite3`, and fall back to python3's own `sqlite3` module when that binary is not installed — so python3 is the one thing to have. Everything else, including `install`, `use` and `verify`, works offline. |
 | Network | only `doctor` and `refresh` need it, and only to read the live model catalogue. `list` is offline: it reads the shipped profiles. Everything else works offline. |
 | Disk | `use` and `install` write into `~/.config/opencode/` and nothing else. `doctor` writes nothing. `refresh` rewrites a profile in the kit's own
 directory - `node_modules` for an npm install - and keeps a backup of the
 one it replaced. |
 
 `~/.config/opencode/` is your live agent config. `use` and `install` write there,
-`verify` only reads, and `use --dryrun` writes nothing at all. Every write keeps
-a `.bak` next to the file, including under `--yes`.
+`verify` only reads, and `use --dryrun` writes nothing at all. Every write
+keeps a dated copy in `~/.config/opencode/backups/`, including under
+`--yes`, and `oc-harness restore <file>` puts one back.
 
 ## Commands
 
@@ -362,6 +370,17 @@ oc-harness baseline   measure the current install
 oc-harness run        drive .space/tasks.md to done with the build agent
 ```
 
+The measurement scripts are in `eval/`, and each one says what it measures
+at the top of the file:
+
+```
+node eval/run.mjs               the pruner and the allowlist
+node eval/measure-prune.mjs     the pruner, on real sessions
+node eval/measure-install.mjs   before and after the install
+node eval/models.mjs            every model the profiles use, one line each
+node eval/ab.mjs                the A/B: the same task with and without the kit
+```
+
 Every flag accepts both spellings: `--profile fleet-safe` and
 `--profile=fleet-safe` are the same flag.
 
@@ -434,18 +453,14 @@ Ten lanes is a hypothesis. `oc-harness lanes` counts the requests:
 ```
 $ oc-harness lanes
 
-  lane            reqs     avg ms  model
-  ----------------------------------------------------------------
-  plan                7       2624  beam-free
-  build              21      10104  minimax-m3.1-flash-free
-  reviewer            2       4716  nemotron-3-ultra-free
-  locate              4       3680  nemotron-3.5-lightning-free
-  verify              2       6546  big-pickle
-  vision              2      19167  mimo-v2.6-flash-free
-  doc                 2       5421  glm-5.3-flash-free
-  schema              1       3390  deepseek-v4.1-flash-free
-  sweep               3       4221  nemotron-3.5-super-vl-free
-  longform            2       9382  longcat-2.5-preview-free
+  lane             reqs     avg ms     avg ctx  model
+  ----------------------------------------------------------------------
+  plan                0          -           -  deepseek-v4.1-flash-free        never used
+  build              87       9935           -  minimax-m3.1-flash-free, deepseek-v4.1-flash-free
+  reviewer            0          -           -  glm-5.3-flash-free              never used
+  ...
+  9 lane(s) were never used cut candidates, not defects
+  one request is one assistant message; avg ms covers the requests that finished
 ```
 
 A lane that is never used is a description the model reads on every turn and
@@ -519,8 +534,10 @@ the number that is.
 parts unpruned — the pruner rewrites the outgoing request, not the history —
 so the stored parts of a real session are exactly what the pruner was given,
 and replaying the real pruner over them measures exactly what it removed, on
-the same messages with and without. On one install: **80.5% of the context
-across 60 real sessions**.
+the same messages with and without. On one install, on 2026-10-20:
+**80.5% of the context across 60 real sessions**. The number moves with the
+sessions, because it is measured over the ones this machine happened to
+have - so run the command for the number on yours.
 
 ## Memory
 
@@ -798,7 +815,8 @@ same 120 runs with the fixed grader gives 34/40 against 30/40 on the lookup
 tasks - the kit ahead, where the old grader had it behind. The full account is in [eval/RESULTS.md](eval/RESULTS.md).
 
 What the kit buys is in the two places that suite cannot see: **long
-sessions**, where the pruner removes 80.5% of the context, and **process** -
+sessions**, where the pruner removes 65-80% of the context depending on the
+sessions it is measured over, and **process** -
 a reviewer that reviews and does not edit, a verify that runs the checks and
 does not fix them, a publish decision that belongs to a human.
 

package/bin/oc-harness.mjs

@@ -1157,7 +1157,7 @@ async function cmdUse(id) {
     // `--yes` skips the question, it does not skip the safety net. Gating the
     // backup on `--yes` meant the one path that overwrites without asking was
     // also the one path with nothing to fall back to.
-    if (!flag("--yes") && interactive() && !(await confirm(`overwrite ${CONFIG_FILE}? (a .bak backup is kept)`))) {
+    if (!flag("--yes") && interactive() && !(await confirm(`overwrite ${CONFIG_FILE}? (a dated backup is kept in backups/)`))) {
       fail("left the existing config alone")
     }
     const kept = keepBackup(readFileSync(CONFIG_FILE, "utf8"))
@@ -1462,7 +1462,7 @@ function applyLiveMetadata(p, live) {
 /**
  * Put back the config the last `use` replaced.
  *
- * `use` keeps one `.bak` next to the config, and it OVERWRITES it on the
+ * `use` keeps a dated copy of the old config in `backups/`, and it OVERWRITES it on the
  * next run. So there is no history, and the backup does not say which
  * profile it was - so "go back to what I had" means "remember the old
  * profile id and run use again", which is not something a person can do at
@@ -2630,6 +2630,9 @@ try {
     --check-models doctor: one run, non-zero when a shipped model is gone
     --json        doctor, show, lanes: print the machine-readable form, and
                   nothing else
+    --hook        install: also install the pre-commit hook into this
+                  project. It is the project's call, so it has to be asked
+                  for.
 
   run options
     --role <agent>  which agent does the work (default: build)

package/docs/EXIT-CODES.md

@@ -1,12 +1,14 @@
 # Exit codes
 
 Every command that can fail says so in its exit code, so a script can tell
-"the kit says no" from "the kit broke". A command that succeeded exits 0 and
-prints nothing to stderr.
+"the kit says no" from "the kit broke". A command that succeeded exits 0.
+Its errors go to stderr; a notice about what it did - a backup it kept, a
+warning about retention - also goes there, so a script that only reads
+stdout gets the answer and nothing else.
 
 ## `oc-harness run`
 
-The unattended loop. It walks `space-tasks.md`, asks the agent for one task
+The unattended loop. It walks `.space/tasks.md`, asks the agent for one task
 at a time, and re-runs the task's own verify command before believing it.
 
 | Code | Meaning |
@@ -15,7 +17,7 @@ at a time, and re-runs the task's own verify command before believing it.
 | 1 | There is no task file. The loop needs a plan to walk. |
 | 2 | Stopped early and a person is needed: a task is blocked, a task has no verify command, the round cap was reached, or the time budget ran out. |
 | 3 | The agent itself failed. The reason is the agent's own last words, and the task is marked blocked with them. |
-| 4 | The verify command failed. The task is marked blocked and the real output is in `space-tasks.md` - not in the exit code, which is no place for a stack trace. |
+| 4 | The verify command failed. The task is marked blocked and the real output is in `.space/tasks.md` - not in the exit code, which is no place for a stack trace. |
 
 Code 2 is the "asked for something impossible" code: nothing about it is a
 bug, and the reason is printed. Codes 3 and 4 mean the work did not land.
@@ -24,11 +26,17 @@ bug, and the reason is printed. Codes 3 and 4 mean the work did not land.
 
 | Code | Meaning |
 |---|---|
-| 0 | Everything checks out. |
-| 1 | Something does not. The problems are listed, one per line. |
+| 0 | The install is consistent, and no profile has drifted. |
+| 1 | The install is broken: no config, no profile recorded in it, a profile that cannot be loaded, or a field that drifted from the profile. The problems are listed, one per line. |
 
-`--check-models` exits 1 as soon as a profile has drifted from the live
-catalogue. `--watch` never exits: it polls, and reports only what CHANGED.
+**A profile that has drifted from the live catalogue is not a broken
+install.** A provider changed a context window; the install is fine and the
+metadata is stale. That exits 0, and the drift is reported. `--strict` is
+the flag that makes it fail, for a CI job that wants to know.
+
+`--check-models` exits 1 when a shipped model has **left** the catalogue -
+not when it has drifted. `--watch` never exits: it polls, and reports only
+what CHANGED.
 
 A failed catalogue fetch is not a finding. It is reported, the models are not
 announced as withdrawn, and the poll carries on - a catalogue that cannot be
@@ -36,7 +44,8 @@ read says nothing about the models in it.
 
 With `--json` the report is the only thing on stdout: a script that pipes it
 into a parser gets JSON, and a failure is a JSON object with an `error` in
-it, not a parse error.
+it, not a parse error. `--json` and `--watch` do not combine: the watch
+prints what changed, in prose, as it happens.
 
 ## The measurement gates
 
@@ -47,6 +56,9 @@ it, not a parse error.
 | 0 | Every measurement that could be compared is inside its threshold. |
 | 1 | One is not. The report names the threshold and the number. |
 
+`node eval/models.mjs` exits 1 when any model could not be measured: a
+table with a hole in it is not a measurement of the tier.
+
 A measurement that could NOT be compared - the two windows do not hold the
 same work, so the difference is confounded - is reported as a NOTE and NOT
 gated. The attributable measurement of the same thing, the pruner replayed
@@ -60,8 +72,8 @@ attributable measurement carries the gate.
 
 | Code | Meaning |
 |---|---|
-| 0 | The install is in place, or the change was confirmed. |
-| 1 | The install was refused, left alone, or a step failed. |
+| 0 | The install is in place, or the change was confirmed. Something may have been left alone - a file that was already there, a prompt that was not answered - and the reason is printed. |
+| 1 | The install was refused, or a step failed. |
 
 `doctor` also exits 1 when there is no config, or no profile recorded in
 it: a kit that is installed but points at nothing is a broken install, and

package/docs/KNOWN-LIMITS.md

@@ -288,7 +288,7 @@ From one install's 20-day history, free model slots turned over in days:
 three days. Zen states free models are available "for a limited time" with no
 date. Treat profiles as perishable.
 
-The README's [Free models are perishable](../../README.md#free-models-are-perishable-so-check-them)
+The README's [Free models are perishable](../README.md#free-models-are-perishable-so-check-them)
 section has the full list of withdrawn models, the four commands that check
 for drift, and what `refresh` will and will not rewrite. The short version:
 run `oc-harness doctor` before every release and whenever a lane starts

package/docs/RESEARCH.md

@@ -193,7 +193,14 @@ out.** Re-measured on 2026-10-20, lane by lane, over live calls: GLM is
 4.4-5.8 seconds and not 16-27 (the old figure came from a much larger
 prompt); MiMo reads an image in 19.2 seconds and not 8; and Nemotron Super VL
 does **not** hang on a tool read, only on a file attached to the prompt, which
-means the sweep lane is sound. The measurement also showed the safe fleet
+means the sweep lane is sound.
+
+Those were the install's numbers, measured inside a working session. A
+DIRECT measurement on 2026-10-21 - one line, the same prompt for every
+model, from the process start - puts GLM at 7.4 seconds and Beam at 6.1, so
+the two are not comparable and the routing page now says which is which.
+The image figures are still the install's: a one-line prompt is not an
+image. The measurement also showed the safe fleet
 giving up more than it needed to: two lanes were on a model the ceiling had
 forced them onto rather than the best one available for that job. The numbers,
 and what changed because of them, are in [`ROUTING.md`](ROUTING.md).
@@ -288,3 +295,6 @@ and quietly replacing it.
   session
 - The live free tier, measured 2026-10-20: one request per lane on both
   fleets, ten lanes on each, through real delegations
+- The live free tier, measured 2026-10-21, directly: `node eval/models.mjs
+  --reps 5 --effort` - the same one-line prompt to every model the profiles
+  use, from the process start, with the `low` variant as well

package/docs/ROUTING.md

@@ -4,9 +4,26 @@ The fleet is the point of this kit. One session, ten lanes, ten models, and a
 human who never switches a model by hand.
 
 This page records what the lanes are, which model serves each one, and the
-measurements behind those choices. Everything in the tables below was measured
-against the live tier on **2026-10-20** with OpenCode **1.19.0**, and is
-reproducible with `oc-harness doctor` and `oc-harness lanes`.
+measurements behind those choices.
+
+There are **two** sets of numbers here, and they measure different things:
+
+- The **install tables** are the telemetry of a real install: `oc-harness
+  lanes` over the requests a working session actually made, on
+  **2026-10-20**. They are the wait a person really had, and the `build` row
+  is not comparable with the others, because it counts the turns that did the
+  delegating.
+- The **direct table** is `node eval/models.mjs` - one line, one model, five
+  requests, whole-process wall time, on **2026-10-21**. It is the same
+  prompt for every model, so it compares models with each other, and it
+  includes the process start, which the install tables do not.
+
+Neither is the answer on its own. The install tables say what happened; the
+direct table says what a lane would wait for if the lane were all there was.
+
+Both are reproducible: `oc-harness doctor` for the catalogue, `oc-harness
+lanes` for the install, and `node eval/models.mjs --reps 5 --effort` for the
+direct table.
 
 ## The lanes
 
@@ -88,11 +105,67 @@ that did the delegating, so they count the work of the other nine lanes. A
 one-line `build` reply on its own measured **3130 ms** and **21137 ms** on a
 cold start.
 
+## Measured directly, one line, one model
+
+    node eval/models.mjs --reps 5 --effort
+
+Every model gets the same prompt - one line, one word back - and the same
+five requests. `wall` is the whole `opencode run`, so it includes the
+process start and the plugin load. `reasoning` is the tokens the model
+spent thinking, which on a prompt this small is close to nothing for most of
+them: the difference between a tier and no tier shows up on real work, not
+on a one-line reply.
+
+| model | wall avg | median | max | reasoning | wall @low | reasoning @low |
+|---|---|---|---|---|---|---|
+| `beam-free` | 7,800 | **6,118** | 13,782 | 24 | 6,308 | 28 |
+| `minimax-m3.1-flash-free` | 7,114 | 6,903 | 8,448 | 0 | **6,292** | 0 |
+| `nemotron-3.5-lightning-free` | 8,140 | 6,882 | 13,554 | 86 | — | — |
+| `glm-5.3-flash-free` | 10,604 | 7,408 | 21,002 | 0 | 10,718 | 12 |
+| `nemotron-3-ultra-free` | 9,282 | 7,847 | 15,402 | 36 | — | — |
+| `nemotron-3.5-super-vl-free` | 10,404 | 8,353 | 19,763 | 15 | 10,988 | 17 |
+| `mimo-v2.6-flash-free` | 8,875 | 9,345 | 9,635 | 8 | — | — |
+| `big-pickle` | 9,983 | 10,438 | 11,372 | 8 | — | — |
+| `deepseek-v4.1-flash-free` | 15,840 | 10,659 | 25,805 | 0 | 14,193 | 46 |
+| `longcat-2.5-preview-free` | 13,099 | 11,996 | 16,991 | 16 | — | — |
+
+**What this says that the install tables do not.** The margin between the
+fastest model and the fourth is under a second - 6.1 s for Beam against 7.4 s
+for GLM - so a plan that has to be quick does not need the fastest model, it
+needs any of the first four. The spread at the other end is real: LongCat
+costs twice what Beam costs, on the same one-line question.
+
+**And the `low` column is not a reason to ask for `low` on a small job.**
+On a prompt this small the difference is inside the noise, and on one of them
+it is the wrong way round. The tier earns its keep on real work, and that
+is the measurement below, on the install's own messages.
+
+A dash in the `@low` columns means the model publishes **no `low` level**,
+so there is nothing to ask for: it runs at its natural rate. The script
+reads the catalogue to know which is which, rather than asking every model
+and reporting an answer it did not use.
+
+The model ids are shown without their provider; the script prints
+`opencode/beam-free`, and either spelling works with `--model`.
+
+**The free tier is noisy, and the table says so.** Five requests is five
+requests: `deepseek` had a 25.8 s outlier in a 10.7 s median, and Beam had a
+13.8 s one in a 6.1 s median. The median is the number to read, and a
+single request is not a measurement of anything.
+
 **What the safe ceiling costs, measured rather than assumed.** Planning is the
-lane that pays: 2624 ms on Beam against 5106 ms on DeepSeek, about twice the
-wait. Reviewing barely pays at all: 4716 ms on Nemotron 3 Ultra against 5757
-ms on GLM. Locating and verifying get *faster*, because DeepSeek publishes a
-`low` effort level and Big Pickle publishes none at all.
+lane that pays: the install measured 2624 ms on Beam against 5106 ms on
+DeepSeek, and the direct measurement says the same thing a different way -
+6.1 s against 10.7 s, about three quarters more.
+
+Reviewing does not pay at all. The install measured 4716 ms on Nemotron 3
+Ultra against 5757 ms on GLM, and the direct measurement has them level, with
+GLM marginally ahead: 7.4 s against 7.8 s. Either way, the model built for
+reviewing is not faster than the one the ceiling puts in its place, and
+neither is slow.
+
+Locating and verifying get *faster*, because DeepSeek publishes a `low` effort
+level and Big Pickle publishes none at all.
 
 **What a tier is worth, measured on this install rather than assumed.** The
 effort level is the cheapest lever there is: DeepSeek measures 320 reasoning
@@ -125,16 +198,48 @@ choice anyway.
 
 | model | measured | why it owns that lane |
 |---|---|---|
-| minimax-m3.1-flash-free | 3.1 s for a one-line reply, 512k output cap, reads an image in 7.7 s | The widest output cap on the tier and a stable tool call, so the lane that writes code fits, and it is also the fastest measured image reader. |
-| beam-free | **2.6 s average over 7 requests, the fastest on the tier**, 1M context, structured output | Planning is a reading-and-reasoning job with a long context and a structured answer. Nothing else on the tier does it faster. |
-| nemotron-3-ultra-free | 4.7 s average, but **34 s on a cold first request** | The largest open MoE available, which is what a two-pass review needs. The cold start is worth knowing about before blaming the route. |
-| nemotron-3.5-lightning-free | 3.7 s average over 4 requests, 5.5 s at worst, no selectable effort | No effort level, so it spends nothing on reasoning, and it can emit a file larger than 128k tokens. |
-| big-pickle | 6.5 s average over 2 requests, structured output, no effort levels, no images | Stable, and it answers in a shape a program can read. |
-| mimo-v2.6-flash-free | **19.2 s average with an image**, 23.8 s at worst - the slowest lane on the tier | It reads images correctly, which is the job. It publishes audio input too, and that path is unreachable (see below). |
-| glm-5.3-flash-free | 4.4-5.8 s, structured output, 3 effort levels | Careful and quick enough. A previous measurement put it at 16-27 s on a larger prompt; on the work a lane actually does it is five seconds. |
-| deepseek-v4.1-flash-free | 3.4-5.2 s, structured output, reasoning + agentic coding | The natural lane for JSON a program has to parse, and the best all-rounder inside a zero-retention ceiling. |
-| nemotron-3.5-super-vl-free | 4.2 s average over 3 requests, **the only model publishing a `none` effort level** | A lane that turns reasoning off entirely is a job of its own, and no other model can do it. |
-| longcat-2.5-preview-free | 9.4-13.2 s, 1M context, reads images, temperature on | The long-context lane. A preview endpoint: it can be withdrawn without a rename. |
+| minimax-m3.1-flash-free | 6.9 s median for a one-line reply, 512k output cap - the widest on the tier, reads an image in 7.7 s | The widest output cap on the tier and a stable tool call, so the lane that writes code fits, and it is also the fastest measured image reader. |
+| beam-free | **6.1 s median, the fastest of the ten**, 1M context, structured output | Planning is a reading-and-reasoning job with a long context and a structured answer. It is the fastest measured, but only by under a second over the next three - so the reason to pick it is the 1M context and the structured output as much as the speed. |
+| nemotron-3-ultra-free | 7.8 s median, but **34 s on a cold first request** | The largest open MoE available, which is what a two-pass review needs. The cold start is worth knowing about before blaming the route. |
+| nemotron-3.5-lightning-free | 6.9 s median, 13.6 s at worst, no selectable effort | No effort level, so it spends nothing on reasoning, and it can emit a file larger than 128k tokens. |
+| big-pickle | 10.4 s median, structured output, no effort levels, no images | Stable, and it answers in a shape a program can read. |
+| mimo-v2.6-flash-free | 9.3 s median for a line of text, **19.2 s average with an image** | It reads images correctly, which is the job. It publishes audio input too, and that path is unreachable (see below). |
+| glm-5.3-flash-free | 7.4 s median, structured output, 3 effort levels | Careful and quick enough. A previous measurement put it at 16-27 s on a larger prompt; on the work a lane actually does it is about seven seconds. |
+| deepseek-v4.1-flash-free | 10.7 s median, structured output, reasoning + agentic coding | The natural lane for JSON a program has to parse, and the best all-rounder inside a zero-retention ceiling. It was the slowest of the ten on the one-line measurement - the tier is free, and free is not fast. |
+| nemotron-3.5-super-vl-free | 8.4 s median, **the only model publishing a `none` effort level** | A lane that turns reasoning off entirely is a job of its own, and no other model can do it. |
+| longcat-2.5-preview-free | 12.0 s median - the slowest of the ten, 1M context, reads images, temperature on | The long-context lane. A preview endpoint: it can be withdrawn without a rename. |
+
+## What each model publishes
+
+From the live catalogue, 2026-10-21. `oc-harness doctor` re-checks all of
+it, and a model that is withdrawn says so rather than being measured.
+
+| model | effort levels | context | output cap | reads |
+|---|---|---|---|---|
+| `beam-free` | low, medium, high, xhigh, max | 1M | 131k | text |
+| `big-pickle` | **none** | 200k | 32k | text |
+| `deepseek-v4.1-flash-free` | low, high, max | 200k | 32k | text, image |
+| `minimax-m3.1-flash-free` | low, medium, high, xhigh, max | 1M | **512k** | text, image, video |
+| `nemotron-3-ultra-free` | **none** | 1M | 128k | text |
+| `nemotron-3.5-lightning-free` | **none** | 262k | **262k** | text |
+| `mimo-v2.6-flash-free` | **none** | 200k | 32k | text, image, audio, video |
+| `glm-5.3-flash-free` | low, high, max | 200k | 32k | text, image, video, pdf |
+| `nemotron-3.5-super-vl-free` | **none**, minimal, low, medium, high, xhigh | 1M | 131k | text, image |
+| `longcat-2.5-preview-free` | **none** | 1M | 131k | text, image |
+
+Two things fall out of this table that the prose above rests on.
+
+**Only one model can turn reasoning off entirely**, and it is
+`nemotron-3.5-super-vl-free`. The `sweep` lane is a bulk mechanical search,
+so reasoning on it is pure waste - and the model that could switch it off
+publishes `none` and the four zero-retention models do not, so under the
+ceiling that lane cannot ask for it. It runs on DeepSeek, which publishes
+`low`, and that is the closest thing available.
+
+**A model that publishes no effort level still spends what it spends.** It
+runs at its natural rate. That is not a defect: it is the reason Big Pickle
+is cheap to use, and the reason the kit asks for a tier only where the tier
+is worth having.
 
 ## What the tier cannot do
 
@@ -232,19 +337,27 @@ is to count the requests:
 ```
 $ oc-harness lanes
 
-  lane            reqs     avg ms  model
-  ----------------------------------------------------------------
-  plan                7       2624  beam-free
-  build              21      10104  minimax-m3.1-flash-free
-  reviewer            2       4716  nemotron-3-ultra-free
+  lane             reqs     avg ms     avg ctx  model
+  ----------------------------------------------------------------------
+  plan                0          -           -  deepseek-v4.1-flash-free        never used
+  build              87       9935           -  minimax-m3.1-flash-free, deepseek-v4.1-flash-free
+  reviewer            0          -           -  glm-5.3-flash-free              never used
   ...
-  8 lane(s) were never used    cut candidates, not defects
+  9 lane(s) were never used cut candidates, not defects
+  one request is one assistant message; avg ms covers the requests that finished
 ```
 
 A lane that is never used is a description the model reads on every turn and a
 free model doing nothing. Cut it from the profile's `roles`, and the lane
 leaves the config with it.
 
+That is also the counter that says a lane is NOT earning its place, and it
+is the honest one to act on: it counts the requests a working session
+actually made. `node eval/models.mjs` answers the other question - how long
+a lane would wait - and the two together are what a routing decision needs.
+The counter alone would cut a lane that is slow but used, and the
+measurement alone would keep a lane nobody calls.
+
 The counter reads `.space/telemetry.jsonl` in the project, written by the
 plugin: one record per assistant message, with the lane, the model and the
 duration. Two things about it are worth knowing, because both were wrong at
@@ -295,5 +408,7 @@ directory is exactly as dangerous as one in the plural one.
   config, so it is sized by the **narrowest** lane: 24,000 tokens for
   `fleet-safe`, whose narrowest lane has a 200k window, and 19,200 for
   `fleet-all`, whose narrowest lane is Big Pickle at a 160k input cap.
+  (Big Pickle's CATALOGUE context is 200k; the profile pins its input cap to
+  160k, and the cap is the number that governs.)
   Sizing it by the widest model instead would reserve 125,829 tokens and
   leave every 200k lane with 74k usable.

package/eval/RESULTS.md

@@ -306,6 +306,12 @@ not a model that answered - four kit-off runs hit the 600-second ceiling
 with no output at all, and a timed-out run has zero tools and zero context,
 so it entered the medians as a fast, empty, wrong answer.
 
+That is why the older result below is quoted as "52/60 against **59/59**":
+one of the sixty kit-off runs produced no answer at all and is excluded
+from every number. Four of the kit-off runs timed out in the later run
+described below, which is why the same suite cannot be compared run to
+run.
+
 | metric (median) | kit on | kit off |
 | --- | --- | --- |
 | **correct** | **54/60** | **55/60** |
@@ -361,10 +367,17 @@ some of the answers took the two-field markdown shape - the shape the
 grader could not read. The kit-off arm had no such instructions, mostly
 answered in prose, and was graded by a pattern that handled prose.
 
-Re-scoring the SAME run with the fixed grader - the same 120 agent runs,
+Re-scoring the EARLIER run with the fixed grader - the same 120 agent runs,
 the same answers, only the grader changed - gives **34/40 against 30/40** on
-the four lookup tasks, the kit ahead. A harness that measures the harness
-is worse than one that measures nothing.
+the four lookup tasks, the kit ahead. The run in the table above, with the
+same grader, scores 34/40 against 35/40 on the same four tasks.
+
+The re-score is reproducible: the raw streams of a run are kept, and
+
+    node eval/ab.mjs --rescore <the stream dir> --target <subject> --suite eval/tasks-self.json
+
+grades them again without spending a token. A harness that measures the
+harness is worse than one that measures nothing.
 
 ### What this does not say
 
@@ -404,7 +417,8 @@ small - only the floor is enforced.
 
 ## Not measured here
 
-- Whether the agent's *answers* got better in general. One task, six runs.
+- Whether the agent's *answers* got better in general. Six tasks, 120 runs,
+  and a tie on correctness - which is not the same as a measurement of it.
 - Whether the numbers in Results 1 and 2 were caused by the kit. Result 3 shows
   a mechanism that would produce them; the A/B points the same way on a
   different task. Neither window separates the kit from the work.

package/eval/measure-install.mjs

@@ -318,6 +318,14 @@ if (failures.length) {
   console.log(`  FAIL  ${failures.join("; ")}`)
 } else if (minNarrow !== null || minCtxDrop !== null) {
   console.log("")
-  console.log(`  PASS  narrow reads and context per request both held.`)
+  // The PASS line has to say WHICH gates held. "Both held" is a lie when
+  // one of them was reported and not gated, and the NOTE above it says so.
+  const held = []
+  if (minNarrow !== null && !skipped.includes("narrow reads")) held.push("narrow reads")
+  if (minCtxDrop !== null && !skipped.includes("context per request")) held.push("context per request")
+  console.log(
+    `  PASS  ${held.length ? held.join(" and ") : "nothing"} held.` +
+      (skipped.length ? ` ${skipped.join(" and ")} was reported but not gated.` : ""),
+  )
 }
 process.exit(failures.length ? 1 : 0)

package/eval/models.mjs

@@ -0,0 +1,313 @@
+#!/usr/bin/env node
+// models.mjs - measure each model on the live tier.
+//
+// The routing table in docs/ROUTING.md says how fast each lane's model is,
+// and a number in a table that nobody can reproduce is a number nobody can
+// check. This is the script that produced the current one.
+//
+//   node eval/models.mjs                 the table, for the models the profiles use
+//   node eval/models.mjs --reps 3        more samples per model
+//   node eval/models.mjs --effort        also measure the `low` effort variant
+//   node eval/models.mjs --json          machine-readable
+//   node eval/models.mjs --model <id>    one model
+//
+// What it measures:
+//
+//   wall ms       the whole `opencode run`, so it includes the process start
+//                 and the round trip. A lane is a round trip, so this is the
+//                 number a person waits for.
+//   reasoning     the tokens the model spent thinking. A model that reasons
+//                 for 400 tokens on a question that needs none is a model
+//                 being asked the wrong way.
+//
+// The prompt is one line and the same for every model, so the comparison is
+// between models and not between prompts. It is not a benchmark of the
+// models: it is a measurement of what a lane would wait for.
+
+import { execFileSync, spawnSync } from "node:child_process"
+import { existsSync, mkdirSync, mkdtempSync, readFileSync, rmSync } from "node:fs"
+import { tmpdir } from "node:os"
+import { join } from "node:path"
+
+import { median } from "./metrics.js"
+import { countFlag, stringFlag } from "./flags.mjs"
+
+const args = process.argv.slice(2)
+const wantJson = args.includes("--json")
+// The shared flag parser, so `--reps=5` means what `--reps 5` means and a
+// mistyped value is an error rather than a measurement of nothing.
+let reps
+let only
+try {
+  reps = countFlag("--reps", args) ?? 3
+  only = stringFlag("--model", args)
+} catch (err) {
+  console.error(err instanceof Error ? err.message : String(err))
+  process.exit(1)
+}
+const withEffort = args.includes("--effort")
+
+const PROMPT = "Reply with exactly the word OK and nothing else."
+
+// The models the shipped profiles use. The catalogue turns over in days, so
+// this is read from the profiles rather than written down here: a model that
+// has been withdrawn is reported as gone instead of measured.
+async function modelsInUse() {
+  const { listProfileIds, loadProfile, modelsInPlay } = await import("../src/profile.js")
+  const seen = new Map()
+  for (const id of listProfileIds()) {
+    let p
+    try {
+      p = loadProfile(id)
+    } catch {
+      continue
+    }
+    for (const m of modelsInPlay(p) ?? []) {
+      // The provider is part of the model id as opencode knows it.
+      const full = m.includes("/") ? m : `${p.provider}/${m}`
+      if (!seen.has(full)) seen.set(full, [])
+      seen.get(full).push(id)
+    }
+  }
+  return [...seen.entries()].map(([model, profiles]) => ({ model, profiles }))
+}
+
+/**
+ * One request to one model.
+ *
+ * @param {string} model
+ * @param {string | null} variant
+ * @returns {{ ok: boolean, ms: number, reasoning: number, output: number, input: number, error: string | null }}
+ */
+function measure(model, variant) {
+  const dir = mkdtempSync(join(tmpdir(), "ock-model-"))
+  const argv = ["run", "--format", "json", "--model", model, "--dir", dir]
+  if (variant) argv.push("--variant", variant)
+  argv.push(PROMPT)
+  const t0 = Date.now()
+  let res
+  try {
+    res = spawnSync("opencode", argv, {
+      cwd: dir,
+      encoding: "utf8",
+      maxBuffer: 64 * 1024 * 1024,
+      timeout: 180_000,
+      killSignal: "SIGKILL",
+    })
+  } finally {
+    // The directory holds a whole opencode project. A run that is killed
+    // leaves it behind, and a scratch directory that is never cleaned up is
+    // a scratch directory nobody can find again.
+    try {
+      rmSync(dir, { recursive: true, force: true })
+    } catch {}
+  }
+  const ms = Date.now() - t0
+  const out = res.stdout ?? ""
+  let reasoning = 0
+  let output = 0
+  let input = 0
+  let finished = false
+  for (const line of out.split("\n")) {
+    if (!line.trim().startsWith("{")) continue
+    let ev
+    try {
+      ev = JSON.parse(line)
+    } catch {
+      continue
+    }
+    if (ev.type !== "step_finish") continue
+    finished = true
+    const t = ev.part?.tokens ?? {}
+    reasoning += Number(t.reasoning ?? 0)
+    output += Number(t.output ?? 0)
+    input += Number(t.input ?? 0) + Number(t.cache?.read ?? 0)
+  }
+  if (!finished) {
+    // The reason, in the order it is useful: the spawn failed (opencode is
+    // not installed), the process was killed (the ceiling), and only then
+    // whatever the provider said. "no reply" on its own says nothing about
+    // which of the three it was.
+    const reason = res.error
+      ? `could not run opencode: ${res.error.message}`
+      : res.signal
+        ? `killed after the ${Math.round(ms / 1000)}s ceiling (${res.signal})`
+        : ((res.stderr ?? "").split("\n").find((l) => l.trim()) ?? "no reply")
+    return { ok: false, ms, reasoning, output, input, error: reason }
+  }
+  return { ok: true, ms, reasoning, output, input, error: null }
+}
+
+/**
+ * The effort levels each model publishes, from the live catalogue.
+ *
+ * A model that publishes no `low` level cannot be asked for one: the
+ * request is ignored and the model runs at its natural rate. Measuring that
+ * and putting it in a column called "wall @low" would say a model is
+ * faster at a tier it does not have.
+ *
+ * @returns {Promise<Map<string, string[]>>}
+ */
+async function publishedEffort() {
+  const out = new Map()
+  const url = process.env.OPENCODE_HARNESS_CATALOG_URL ?? "https://models.dev/api.json"
+  try {
+    const body = url.startsWith("file://")
+      ? readFileSync(url.slice("file://".length), "utf8")
+      : execFileSync("curl", ["-sL", "--fail", "--max-time", "40", url], {
+          encoding: "utf8",
+          // The catalogue is over a megabyte, and the default buffer is one
+          // megabyte: without this the fetch fails with ENOBUFS, the catch
+          // swallows it, and every model looks like it publishes no effort
+          // level at all.
+          maxBuffer: 64 * 1024 * 1024,
+        })
+    const catalog = JSON.parse(body)
+    for (const [provider, p] of Object.entries(catalog ?? {})) {
+      for (const [id, m] of Object.entries(p?.models ?? {})) {
+        const levels = []
+        for (const ro of m?.reasoning_options ?? []) {
+          if (ro?.type === "effort") levels.push(...(ro?.values ?? []))
+        }
+        out.set(`${provider}/${id}`, levels)
+      }
+    }
+  } catch (err) {
+    // A catalogue that cannot be read is not a reason to refuse to measure.
+    // The levels are used to mark a column, not to decide anything - but the
+    // reason is printed, because a silent failure here marks every model as
+    // publishing no effort level, which is a false statement about them.
+    console.error(
+      `  could not read the catalogue: ${err instanceof Error ? err.message : String(err)}`,
+    )
+    console.error("  the effort levels are unknown, so the @low column will read n/a.")
+  }
+  return out
+}
+
+const effort = await publishedEffort()
+
+const rows = []
+for (const { model, profiles } of await modelsInUse()) {
+  // The docs name a model by its bare id, so both spellings work.
+  if (only && model !== only && model !== `opencode/${only}` && only !== model.split("/").pop()) continue
+  const samples = []
+  for (let i = 0; i < reps; i++) {
+    let r = measure(model, null)
+    // One retry. The free tier drops requests, and a model that failed once
+    // is not a model that is gone - reporting it as one would hide the
+    // difference between "withdrawn" and "the provider was busy".
+    if (!r.ok) r = measure(model, null)
+    if (!r.ok) {
+      samples.length = 0
+      rows.push({ model, profiles, ok: false, error: r.error })
+      break
+    }
+    samples.push(r)
+  }
+  if (!samples.length) continue
+  const ms = samples.map((s) => s.ms)
+  // The `low` variant is only measured for a model that PUBLISHES `low`.
+  // A model with no effort levels runs at its natural rate whatever is
+  // asked for, and a column that calls that "@low" is a lie.
+  const levels = effort.get(model) ?? []
+  const publishesLow = levels.includes("low")
+  const low = []
+  if (withEffort && publishesLow) {
+    for (let i = 0; i < reps; i++) {
+      const r = measure(model, "low")
+      if (r.ok) low.push(r)
+    }
+  }
+  rows.push({
+    model,
+    profiles,
+    ok: true,
+    requests: samples.length,
+    avg_ms: Math.round(ms.reduce((a, b) => a + b, 0) / ms.length),
+    median_ms: Math.round(median(ms)),
+    max_ms: Math.max(...ms),
+    reasoning: Math.round(median(samples.map((s) => s.reasoning))),
+    output: Math.round(median(samples.map((s) => s.output))),
+    input: Math.round(median(samples.map((s) => s.input))),
+    effort_levels: levels,
+    publishes_low: publishesLow,
+    low_requests: low.length,
+    low_reasoning: low.length ? Math.round(median(low.map((r) => r.reasoning))) : null,
+    low_avg_ms: low.length ? Math.round(low.reduce((a, r) => a + r.ms, 0) / low.length) : null,
+  })
+  process.stderr.write(
+    `  measured ${model} (${samples.length} request(s), median ${rows.at(-1).median_ms} ms)\n`,
+  )
+}
+
+const failed = rows.filter((r) => !r.ok).length
+if (wantJson) {
+  console.log(
+    JSON.stringify(
+      { when: new Date().toISOString(), reps, prompt: PROMPT, ok: failed === 0, failed, rows },
+      null,
+      2,
+    ),
+  )
+  process.exit(failed === 0 ? 0 : 1)
+}
+
+console.log("")
+console.log(`MEASURED ON THE LIVE TIER  ${new Date().toISOString().slice(0, 10)}`)
+console.log(`  ${reps} request(s) per model, one-line prompt, whole-process wall time`)
+console.log("")
+const head = ["model", "reqs", "avg ms", "median", "max", "reasoning", "output"]
+console.log("  " + head.map((h, i) => (i === 0 ? h.padEnd(30) : h.padStart(10))).join(""))
+console.log("  " + "-".repeat(30 + 10 * (head.length - 1)))
+for (const r of rows) {
+  if (!r.ok) {
+    console.log("  " + r.model.padEnd(30) + `  FAILED: ${String(r.error).slice(0, 70)}`)
+    continue
+  }
+  console.log(
+    "  " +
+      r.model.padEnd(30) +
+      String(r.requests).padStart(10) +
+      String(r.avg_ms).padStart(10) +
+      String(r.median_ms).padStart(10) +
+      String(r.max_ms).padStart(10) +
+      String(r.reasoning).padStart(10) +
+      String(r.output).padStart(10),
+  )
+  if (r.publishes_low) {
+    if (!r.low_requests) {
+      // The model publishes `low` and every request for it failed. Saying
+      // "0 null" reads like a measurement of nothing; saying it failed says
+      // what happened.
+      console.log(
+        "  " +
+          `${r.model} @low`.padEnd(30) +
+          "no request for the low variant succeeded".padStart(10),
+      )
+    } else {
+      console.log(
+        "  " +
+          `${r.model} @low`.padEnd(30) +
+          String(r.low_requests).padStart(10) +
+          String(r.low_avg_ms).padStart(10) +
+          "".padStart(10) +
+          "".padStart(10) +
+          String(r.low_reasoning).padStart(10),
+      )
+    }
+  } else if (withEffort) {
+    console.log(
+      "  " +
+        `${r.model} @low`.padEnd(30) +
+        "n/a".padStart(10) +
+        "  (publishes no low level - it runs at its natural rate)",
+    )
+  }
+}
+console.log("")
+
+if (failed) {
+  console.error(`  ${failed} model(s) could not be measured - the numbers above are not the whole tier.`)
+  process.exit(1)
+}

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "opencode-harness-kit",
-  "version": "0.4.4",
+  "version": "0.4.6",
   "description": "Harness kit for OpenCode: capability profiles, a fleet of job lanes with one free model each, role-based reasoning effort, context pruning, and measured baselines.",
   "license": "MIT",
   "homepage": "https://github.com/pham-canh/space-harness#readme",