opencode-harness-kit 0.4.3 → 0.4.5

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (74) hide show
  1. package/CHANGELOG.md +569 -18
  2. package/CONTRIBUTING.md +249 -0
  3. package/GLOBAL.md +34 -5
  4. package/MIGRATION.md +151 -0
  5. package/README.md +354 -66
  6. package/SECURITY.md +30 -0
  7. package/bin/baseline.mjs +15 -7
  8. package/bin/git-hooks/pre-commit +41 -9
  9. package/bin/oc-harness.mjs +1268 -82
  10. package/dist/config.d.ts +127 -0
  11. package/dist/denylist.d.ts +33 -0
  12. package/dist/effort.d.ts +64 -0
  13. package/dist/guard.d.ts +23 -0
  14. package/dist/health.d.ts +43 -0
  15. package/dist/index.d.ts +9 -0
  16. package/dist/index.js +532 -149
  17. package/dist/plugin.d.ts +15 -0
  18. package/dist/profile.d.ts +399 -0
  19. package/dist/project.d.ts +8 -0
  20. package/dist/prune.d.ts +6 -0
  21. package/dist/router.d.ts +3 -0
  22. package/dist/rules.d.ts +109 -0
  23. package/dist/safe.d.ts +7 -0
  24. package/dist/telemetry-store.d.ts +12 -0
  25. package/dist/telemetry.d.ts +3 -0
  26. package/docs/EXIT-CODES.md +83 -0
  27. package/docs/KNOWN-LIMITS.md +45 -21
  28. package/docs/RESEARCH.md +20 -5
  29. package/docs/ROUTING.md +191 -31
  30. package/eval/RESULTS.md +457 -0
  31. package/eval/ab.mjs +743 -0
  32. package/eval/attribute.mjs +170 -0
  33. package/eval/flags.mjs +88 -0
  34. package/eval/measure-install.mjs +331 -0
  35. package/eval/measure-prune.mjs +339 -0
  36. package/eval/metrics.js +79 -0
  37. package/eval/models.mjs +302 -0
  38. package/eval/premises.mjs +200 -0
  39. package/eval/prepare.mjs +233 -0
  40. package/eval/run.mjs +376 -0
  41. package/eval/tasks-self.json +81 -0
  42. package/eval/tasks-ts.json +50 -0
  43. package/eval/tasks.json +83 -0
  44. package/eval/thresholds.mjs +134 -0
  45. package/eval/verify.mjs +287 -0
  46. package/package.json +44 -17
  47. package/profiles/fleet-all.json +6 -1
  48. package/profiles/fleet-safe.json +7 -3
  49. package/profiles/schema.json +182 -51
  50. package/prompts/build.md +4 -2
  51. package/skills/space-build/SKILL.md +4 -2
  52. package/skills/space-finish/SKILL.md +6 -2
  53. package/skills/space-plan/reference/task-template.md +13 -5
  54. package/src/config.js +182 -32
  55. package/src/denylist.js +95 -0
  56. package/src/driver/run.mjs +81 -16
  57. package/src/effort.js +15 -4
  58. package/src/guard.ts +165 -13
  59. package/src/health.js +80 -0
  60. package/src/index.ts +15 -4
  61. package/src/lane-usage.js +72 -18
  62. package/src/plugin-type-check.ts +38 -0
  63. package/src/plugin.ts +39 -0
  64. package/src/profile.js +225 -26
  65. package/src/project.js +47 -0
  66. package/src/prune.ts +110 -31
  67. package/src/router.ts +32 -63
  68. package/src/rules.js +99 -7
  69. package/src/safe.ts +27 -0
  70. package/src/sqlite.js +144 -0
  71. package/src/tasks.js +112 -29
  72. package/src/telemetry-store.ts +62 -0
  73. package/src/telemetry.ts +60 -57
  74. package/src/trust.js +70 -9
@@ -96,8 +96,8 @@ a client cannot currently tell a short quota window from a permanent block.
96
96
 
97
97
  ## A copied plugin loads and does nothing
98
98
 
99
- The bundle finds `profiles/`, `prompts/`, `retention.yml` and `lanes.json` by
100
- walking up from its own file. A symlink resolves to the real path, so a
99
+ The bundle finds `profiles/` and `lanes.json` by walking up from its own file,
100
+ and `retention.yml` next to them. A symlink resolves to the real path, so a
101
101
  symlinked plugin finds the kit's data. A **copy** does not: it looks beside
102
102
  itself, finds nothing, and every profile lookup fails.
103
103
 
@@ -284,23 +284,47 @@ unhealthy and failing over at the next start is the honest behaviour.
284
284
  ## Free models get withdrawn quickly
285
285
 
286
286
  From one install's 20-day history, free model slots turned over in days:
287
-
288
- | model | observed lifetime |
289
- |---|---|
290
- | `mimo-v2.5-free` | 1 day |
291
- | `big-pickle` | 1 day |
292
- | `nemotron-3-ultra-free` | 3 days |
293
- | `muse-spark-1.3-contributor-free` | still running at 20 days |
294
-
295
- The tier turned over again since: four of this kit's five original profiles
296
- (`space-bunny-free`, `x-preview-f-free`, `deepseek-v4-flash-free`,
297
- `muse-spark-1.3-contributor-free`) were all withdrawn within a month of
298
- shipping, and the default profile pointed at a model that no longer existed.
299
-
300
- Zen states free models are available "for a limited time" with no date. Treat
301
- profiles as perishable: run `oc-harness doctor`, which re-checks each profile
302
- against live metadata and reports drift, and which also reports a lane whose
303
- whole chain has left the catalogue.
287
+ `mimo-v2.5-free` lasted a day, `big-pickle` a day, `nemotron-3-ultra-free`
288
+ three days. Zen states free models are available "for a limited time" with no
289
+ date. Treat profiles as perishable.
290
+
291
+ The README's [Free models are perishable](../README.md#free-models-are-perishable-so-check-them)
292
+ section has the full list of withdrawn models, the four commands that check
293
+ for drift, and what `refresh` will and will not rewrite. The short version:
294
+ run `oc-harness doctor` before every release and whenever a lane starts
295
+ failing with a 429 or a withdrawal.
296
+
297
+ The tier turned over again since. Every one of this kit's original
298
+ profiles has been withdrawn — `space-bunny-free`, `x-preview-f-free`,
299
+ `deepseek-v4-flash-free`, `muse-spark-1.3-contributor-free` and
300
+ `mimo-v2.5-free` — and the default profile pointed at a model that no longer
301
+ existed.
302
+
303
+ ## The install-wide measurement is an observation, and there is a controlled one
304
+
305
+ `npm run measure` compares the work before the kit landed with the work
306
+ after. It reported the context per request going **up** by 16% on one install,
307
+ with twice as many reads and twice as many messages in the second window.
308
+
309
+ That is a number about the jobs, not about the kit. The two windows are never
310
+ the same work, and "everything since the install" is a window that grows
311
+ without bound, so the comparison is a different one every time the script
312
+ runs.
313
+
314
+ `npm run measure` now says so: the windows are fixed and equal around the
315
+ split, the context drop is skipped with a note when the two sides are not
316
+ comparable, and the gate uses the number that IS comparable.
317
+
318
+ `eval/measure-prune.mjs` is that number. opencode stores the message parts
319
+ unpruned - the pruner rewrites the outgoing request, not the history - so the
320
+ stored parts of a real session are exactly what the pruner was given, and
321
+ replaying the real pruner over them measures exactly what it removed. On one
322
+ install: **80.5% of the context
323
+ across 60 real sessions**, 6,608 parts stubbed - the number
324
+ `npm run measure:prune` prints.
325
+
326
+ The controlled comparison of the two arms is `eval/ab.mjs`, and its result is
327
+ in `eval/RESULTS.md`.
304
328
 
305
329
  ## A model-driven compressor is not assumed to work
306
330
 
@@ -322,7 +346,7 @@ truncation directory, so re-reading is possible. But "re-run the command to see
322
346
  it again" is more expensive than "scroll up". The stub is one line precisely so
323
347
  an agent can decide whether it is worth it, and the stub says so.
324
348
 
325
- A `savedBytes` of 0 in `prune.jsonl` is not a second, redundant pass. It means
349
+ A `savedChars` of 0 in `prune.jsonl` is not a second, redundant pass. It means
326
350
  `bytesAfter` did not come out below `bytesBefore` for that turn, which happens
327
351
  when a stub is no shorter than the output it replaced (the stub's own label
328
352
  costs more than a short body), or when a `reasoning` part was under the 200
@@ -346,7 +370,7 @@ compressor, or a session UI. Those exist and are good:
346
370
  |---|---|
347
371
  | model-driven compression | `@tarquinen/opencode-dcp` |
348
372
  | cross-harness context proxy | `sleev.ai` |
349
- | persistent memory | `opencode-supermemory` |
373
+ | persistent memory | [`opencode-mem`](https://github.com/tickernelz/opencode-mem), which `oc-harness use --memory` turns on |
350
374
  | git worktrees | `kdcokenny/opencode-worktree` |
351
375
  | memory in git markdown | `code-yeongyu/oh-my-opencode` (now `omo`) |
352
376
  | agent DAGs | `spoons-and-mirrors/subtask2` |
package/docs/RESEARCH.md CHANGED
@@ -47,13 +47,18 @@ ceiling that a normal session never reaches.
47
47
 
48
48
  5,239 historical `bash` commands were replayed against the allowlist shapes:
49
49
 
50
- | allowlist | prompts | share |
50
+ | allowlist | prompts | share of 5,239 |
51
51
  |---|---|---|
52
52
  | the narrow one in use | 4,994 | 95.3% |
53
53
  | this kit's, first version | 1,204 | 23.0% |
54
- | this kit's, current | 1,002 | 18.5% |
54
+ | this kit's, current | 1,002 | 19.1% |
55
55
 
56
- The remaining 18.5% is the deliberately gated set, ranked by how often it
56
+ The three rows share the denominator, and the arithmetic is theirs to be
57
+ checked: 4,994/5,239 is 95.3%, 1,204/5,239 is 23.0%, and 1,002/5,239 is
58
+ 19.1%. The last figure was quoted as 18.5% in two places, which is a
59
+ different number for the same count.
60
+
61
+ The remaining 19.1% is the deliberately gated set, ranked by how often it
57
62
  actually came up:
58
63
 
59
64
  ```
@@ -111,7 +116,7 @@ is what actually bounds context.
111
116
  `bin/baseline.mjs` now prints all five numbers so this is re-measurable instead
112
117
  of being re-argued.
113
118
 
114
- **"Compaction never ran, therefore nothing is wrong."** Zero events across 156
119
+ **"Compaction never ran, therefore nothing is wrong."** Zero events across 153
115
120
  sessions was read as "the setting is fine" rather than "nothing is being
116
121
  bounded". Both readings were half right and the compound conclusion was wrong:
117
122
  the configured `tail_turns` was inert *and* there was no bound on growth.
@@ -188,7 +193,14 @@ out.** Re-measured on 2026-10-20, lane by lane, over live calls: GLM is
188
193
  4.4-5.8 seconds and not 16-27 (the old figure came from a much larger
189
194
  prompt); MiMo reads an image in 19.2 seconds and not 8; and Nemotron Super VL
190
195
  does **not** hang on a tool read, only on a file attached to the prompt, which
191
- means the sweep lane is sound. The measurement also showed the safe fleet
196
+ means the sweep lane is sound.
197
+
198
+ Those were the install's numbers, measured inside a working session. A
199
+ DIRECT measurement on 2026-10-21 - one line, the same prompt for every
200
+ model, from the process start - puts GLM at 7.4 seconds and Beam at 6.1, so
201
+ the two are not comparable and the routing page now says which is which.
202
+ The image figures are still the install's: a one-line prompt is not an
203
+ image. The measurement also showed the safe fleet
192
204
  giving up more than it needed to: two lanes were on a model the ceiling had
193
205
  forced them onto rather than the best one available for that job. The numbers,
194
206
  and what changed because of them, are in [`ROUTING.md`](ROUTING.md).
@@ -283,3 +295,6 @@ and quietly replacing it.
283
295
  session
284
296
  - The live free tier, measured 2026-10-20: one request per lane on both
285
297
  fleets, ten lanes on each, through real delegations
298
+ - The live free tier, measured 2026-10-21, directly: `node eval/models.mjs
299
+ --reps 5 --effort` - the same one-line prompt to every model the profiles
300
+ use, from the process start, with the `low` variant as well
package/docs/ROUTING.md CHANGED
@@ -4,9 +4,26 @@ The fleet is the point of this kit. One session, ten lanes, ten models, and a
4
4
  human who never switches a model by hand.
5
5
 
6
6
  This page records what the lanes are, which model serves each one, and the
7
- measurements behind those choices. Everything in the tables below was measured
8
- against the live tier on **2026-10-20** with OpenCode **1.19.0**, and is
9
- reproducible with `oc-harness doctor` and `oc-harness lanes`.
7
+ measurements behind those choices.
8
+
9
+ There are **two** sets of numbers here, and they measure different things:
10
+
11
+ - The **install tables** are the telemetry of a real install: `oc-harness
12
+ lanes` over the requests a working session actually made, on
13
+ **2026-10-20**. They are the wait a person really had, and the `build` row
14
+ is not comparable with the others, because it counts the turns that did the
15
+ delegating.
16
+ - The **direct table** is `node eval/models.mjs` - one line, one model, five
17
+ requests, whole-process wall time, on **2026-10-21**. It is the same
18
+ prompt for every model, so it compares models with each other, and it
19
+ includes the process start, which the install tables do not.
20
+
21
+ Neither is the answer on its own. The install tables say what happened; the
22
+ direct table says what a lane would wait for if the lane were all there was.
23
+
24
+ Both are reproducible: `oc-harness doctor` for the catalogue, `oc-harness
25
+ lanes` for the install, and `node eval/models.mjs --reps 5 --effort` for the
26
+ direct table.
10
27
 
11
28
  ## The lanes
12
29
 
@@ -23,7 +40,7 @@ models. A profile says which lanes it enables and which model serves each one.
23
40
  | `vision` | read an image or a screenshot | mimo-v2.6-flash-free | minimax-m3.1-flash-free |
24
41
  | `doc` | read a long document or a spec | glm-5.3-flash-free | glm-5.3-flash-free |
25
42
  | `schema` | produce JSON or a JSON schema | deepseek-v4.1-flash-free | deepseek-v4.1-flash-free |
26
- | `sweep` | sweep a lot of code mechanically | nemotron-3.5-super-vl-free | longcat-2.5-preview-free |
43
+ | `sweep` | sweep a lot of code mechanically | nemotron-3.5-super-vl-free | deepseek-v4.1-flash-free |
27
44
  | `longform` | read a megabyte, or write alternatives | longcat-2.5-preview-free | longcat-2.5-preview-free |
28
45
 
29
46
  `fleet-all` gives every lane the model built for it: ten jobs, ten models,
@@ -35,6 +52,17 @@ ceiling, and it is worth paying only when the model that takes over is the
35
52
  best one available *for that job*, which is why the two fleets order their
36
53
  chains differently (see below).
37
54
 
55
+ The `sweep` lane is the one that was worth changing rather than accepting.
56
+ A bulk mechanical search is the one job where reasoning is pure waste, and
57
+ the model it ran on reasoned 420 tokens a message at 16.6 seconds - the
58
+ most expensive way to ask a model to grep. It runs on DeepSeek now, which is
59
+ zero-retention, publishes a `low` level, and measures 19 reasoning tokens at
60
+ 3.4 seconds.
61
+
62
+ Under that ceiling every lane that asks for a tier GETS one. The safe fleet
63
+ has no lane with a dropped tier request; the unrestricted fleet has three, and
64
+ the measurement says they are the three cheapest reasoners on the tier.
65
+
38
66
  ## Measured, per lane
39
67
 
40
68
  Every number here is a live call, not a catalog reading. A **request** is one
@@ -77,11 +105,88 @@ that did the delegating, so they count the work of the other nine lanes. A
77
105
  one-line `build` reply on its own measured **3130 ms** and **21137 ms** on a
78
106
  cold start.
79
107
 
108
+ ## Measured directly, one line, one model
109
+
110
+ node eval/models.mjs --reps 5 --effort
111
+
112
+ Every model gets the same prompt - one line, one word back - and the same
113
+ five requests. `wall` is the whole `opencode run`, so it includes the
114
+ process start and the plugin load. `reasoning` is the tokens the model
115
+ spent thinking, which on a prompt this small is close to nothing for most of
116
+ them: the difference between a tier and no tier shows up on real work, not
117
+ on a one-line reply.
118
+
119
+ | model | wall avg | median | max | reasoning | wall @low | reasoning @low |
120
+ |---|---|---|---|---|---|---|
121
+ | `beam-free` | 7,800 | **6,118** | 13,782 | 24 | 6,308 | 28 |
122
+ | `minimax-m3.1-flash-free` | 7,114 | 6,903 | 8,448 | 0 | **6,292** | 0 |
123
+ | `nemotron-3.5-lightning-free` | 8,140 | 6,882 | 13,554 | 86 | — | — |
124
+ | `glm-5.3-flash-free` | 10,604 | 7,408 | 21,002 | 0 | 10,718 | 12 |
125
+ | `nemotron-3-ultra-free` | 9,282 | 7,847 | 15,402 | 36 | — | — |
126
+ | `nemotron-3.5-super-vl-free` | 10,404 | 8,353 | 19,763 | 15 | 10,988 | 17 |
127
+ | `mimo-v2.6-flash-free` | 8,875 | 9,345 | 9,635 | 8 | — | — |
128
+ | `big-pickle` | 9,983 | 10,438 | 11,372 | 8 | — | — |
129
+ | `deepseek-v4.1-flash-free` | 15,840 | 10,659 | 25,805 | 0 | 14,193 | 46 |
130
+ | `longcat-2.5-preview-free` | 13,099 | 11,996 | 16,991 | 16 | — | — |
131
+
132
+ **What this says that the install tables do not.** The margin between the
133
+ fastest model and the fourth is under a second - 6.1 s for Beam against 7.4 s
134
+ for GLM - so a plan that has to be quick does not need the fastest model, it
135
+ needs any of the first four. The spread at the other end is real: LongCat
136
+ costs twice what Beam costs, on the same one-line question.
137
+
138
+ **And the `low` column is not a reason to ask for `low` on a small job.**
139
+ On a prompt this small the difference is inside the noise, and on one of them
140
+ it is the wrong way round. The tier earns its keep on real work, and that
141
+ is the measurement below, on the install's own messages.
142
+
143
+ A dash in the `@low` columns means the model publishes **no `low` level**,
144
+ so there is nothing to ask for: it runs at its natural rate. The script
145
+ reads the catalogue to know which is which, rather than asking every model
146
+ and reporting an answer it did not use.
147
+
148
+ The model ids are shown without their provider; the script prints
149
+ `opencode/beam-free`, and either spelling works with `--model`.
150
+
151
+ **The free tier is noisy, and the table says so.** Five requests is five
152
+ requests: `deepseek` had a 25.8 s outlier in a 10.7 s median, and Beam had a
153
+ 13.8 s one in a 6.1 s median. The median is the number to read, and a
154
+ single request is not a measurement of anything.
155
+
80
156
  **What the safe ceiling costs, measured rather than assumed.** Planning is the
81
- lane that pays: 2624 ms on Beam against 5106 ms on DeepSeek, about twice the
82
- wait. Reviewing barely pays at all: 4716 ms on Nemotron 3 Ultra against 5757
83
- ms on GLM. Locating and verifying get *faster*, because DeepSeek publishes a
84
- `low` effort level and Big Pickle publishes none at all.
157
+ lane that pays: the install measured 2624 ms on Beam against 5106 ms on
158
+ DeepSeek, and the direct measurement says the same thing a different way -
159
+ 6.1 s against 10.7 s, about three quarters more.
160
+
161
+ Reviewing does not pay at all. The install measured 4716 ms on Nemotron 3
162
+ Ultra against 5757 ms on GLM, and the direct measurement has them level, with
163
+ GLM marginally ahead: 7.4 s against 7.8 s. Either way, the model built for
164
+ reviewing is not faster than the one the ceiling puts in its place, and
165
+ neither is slow.
166
+
167
+ Locating and verifying get *faster*, because DeepSeek publishes a `low` effort
168
+ level and Big Pickle publishes none at all.
169
+
170
+ **What a tier is worth, measured on this install rather than assumed.** The
171
+ effort level is the cheapest lever there is: DeepSeek measures 320 reasoning
172
+ tokens a message at `max` and 19 at `low`, a factor of 17, and a lane that
173
+ asks for a tier and lands on a model that publishes none runs at that model's
174
+ NATURAL rate instead. So the tier request is worth protecting.
175
+
176
+ The three lanes in `fleet-all` whose model cannot serve the tier - `reviewer`
177
+ on Nemotron 3 Ultra, `locate` on Nemotron 3.5 Lightning, `verify` on Big
178
+ Pickle - are the fleet's CHEAPEST reasoners: 116, 59 and 18 reasoning tokens a
179
+ message, against 289 for `build` and 198 for `plan`, which keep their tier on
180
+ purpose. The tier is the lever, and it is not where the tokens are. The
181
+ config header says so where it happens, in the form `locate=unset(asked low)`.
182
+
183
+ One lane was not cheap, and it was fixed. `sweep` is a bulk mechanical
184
+ search, so reasoning on it is pure waste, and it ran on LongCat: 420 reasoning
185
+ tokens a message at 16.6 s, the most expensive way to ask a model to grep. No
186
+ zero-retention model publishes a `none` level, so the lane could not even turn
187
+ reasoning off when it asked to. It runs on DeepSeek now, which is
188
+ zero-retention, publishes a `low` level, and measures 19 reasoning tokens at
189
+ 3.4 s.
85
190
 
86
191
  And one lane gets faster rather than slower. MiMo reads an image in 19.2 s;
87
192
  MiniMax reads the same image in 7.7 s. The unrestricted fleet put `vision` on
@@ -93,16 +198,48 @@ choice anyway.
93
198
 
94
199
  | model | measured | why it owns that lane |
95
200
  |---|---|---|
96
- | minimax-m3.1-flash-free | 3.1 s for a one-line reply, 512k output cap, reads an image in 7.7 s | The widest output cap on the tier and a stable tool call, so the lane that writes code fits, and it is also the fastest measured image reader. |
97
- | beam-free | **2.6 s average over 7 requests, the fastest on the tier**, 1M context, structured output | Planning is a reading-and-reasoning job with a long context and a structured answer. Nothing else on the tier does it faster. |
98
- | nemotron-3-ultra-free | 4.7 s average, but **34 s on a cold first request** | The largest open MoE available, which is what a two-pass review needs. The cold start is worth knowing about before blaming the route. |
99
- | nemotron-3.5-lightning-free | 3.7 s average over 4 requests, 5.5 s at worst, no selectable effort | No effort level, so it spends nothing on reasoning, and it can emit a file larger than 128k tokens. |
100
- | big-pickle | 6.5 s average over 2 requests, structured output, no effort levels, no images | Stable, and it answers in a shape a program can read. |
101
- | mimo-v2.6-flash-free | **19.2 s average with an image**, 23.8 s at worst - the slowest lane on the tier | It reads images correctly, which is the job. It publishes audio input too, and that path is unreachable (see below). |
102
- | glm-5.3-flash-free | 4.4-5.8 s, structured output, 3 effort levels | Careful and quick enough. A previous measurement put it at 16-27 s on a larger prompt; on the work a lane actually does it is five seconds. |
103
- | deepseek-v4.1-flash-free | 3.4-5.2 s, structured output, reasoning + agentic coding | The natural lane for JSON a program has to parse, and the best all-rounder inside a zero-retention ceiling. |
104
- | nemotron-3.5-super-vl-free | 4.2 s average over 3 requests, **the only model publishing a `none` effort level** | A lane that turns reasoning off entirely is a job of its own, and no other model can do it. |
105
- | longcat-2.5-preview-free | 9.4-13.2 s, 1M context, reads images, temperature on | The long-context lane. A preview endpoint: it can be withdrawn without a rename. |
201
+ | minimax-m3.1-flash-free | 6.9 s median for a one-line reply, 512k output cap - the widest on the tier, reads an image in 7.7 s | The widest output cap on the tier and a stable tool call, so the lane that writes code fits, and it is also the fastest measured image reader. |
202
+ | beam-free | **6.1 s median, the fastest of the ten**, 1M context, structured output | Planning is a reading-and-reasoning job with a long context and a structured answer. It is the fastest measured, but only by under a second over the next three - so the reason to pick it is the 1M context and the structured output as much as the speed. |
203
+ | nemotron-3-ultra-free | 7.8 s median, but **34 s on a cold first request** | The largest open MoE available, which is what a two-pass review needs. The cold start is worth knowing about before blaming the route. |
204
+ | nemotron-3.5-lightning-free | 6.9 s median, 13.6 s at worst, no selectable effort | No effort level, so it spends nothing on reasoning, and it can emit a file larger than 128k tokens. |
205
+ | big-pickle | 10.4 s median, structured output, no effort levels, no images | Stable, and it answers in a shape a program can read. |
206
+ | mimo-v2.6-flash-free | 9.3 s median for a line of text, **19.2 s average with an image** | It reads images correctly, which is the job. It publishes audio input too, and that path is unreachable (see below). |
207
+ | glm-5.3-flash-free | 7.4 s median, structured output, 3 effort levels | Careful and quick enough. A previous measurement put it at 16-27 s on a larger prompt; on the work a lane actually does it is about seven seconds. |
208
+ | deepseek-v4.1-flash-free | 10.7 s median, structured output, reasoning + agentic coding | The natural lane for JSON a program has to parse, and the best all-rounder inside a zero-retention ceiling. It was the slowest of the ten on the one-line measurement - the tier is free, and free is not fast. |
209
+ | nemotron-3.5-super-vl-free | 8.4 s median, **the only model publishing a `none` effort level** | A lane that turns reasoning off entirely is a job of its own, and no other model can do it. |
210
+ | longcat-2.5-preview-free | 12.0 s median - the slowest of the ten, 1M context, reads images, temperature on | The long-context lane. A preview endpoint: it can be withdrawn without a rename. |
211
+
212
+ ## What each model publishes
213
+
214
+ From the live catalogue, 2026-10-21. `oc-harness doctor` re-checks all of
215
+ it, and a model that is withdrawn says so rather than being measured.
216
+
217
+ | model | effort levels | context | output cap | reads |
218
+ |---|---|---|---|---|
219
+ | `beam-free` | low, medium, high, xhigh, max | 1M | 131k | text |
220
+ | `big-pickle` | **none** | 200k | 32k | text |
221
+ | `deepseek-v4.1-flash-free` | low, high, max | 200k | 32k | text, image |
222
+ | `minimax-m3.1-flash-free` | low, medium, high, xhigh, max | 1M | **512k** | text, image, video |
223
+ | `nemotron-3-ultra-free` | **none** | 1M | 128k | text |
224
+ | `nemotron-3.5-lightning-free` | **none** | 262k | **262k** | text |
225
+ | `mimo-v2.6-flash-free` | **none** | 200k | 32k | text, image, audio, video |
226
+ | `glm-5.3-flash-free` | low, high, max | 200k | 32k | text, image, video, pdf |
227
+ | `nemotron-3.5-super-vl-free` | **none**, minimal, low, medium, high, xhigh | 1M | 131k | text, image |
228
+ | `longcat-2.5-preview-free` | **none** | 1M | 131k | text, image |
229
+
230
+ Two things fall out of this table that the prose above rests on.
231
+
232
+ **Only one model can turn reasoning off entirely**, and it is
233
+ `nemotron-3.5-super-vl-free`. The `sweep` lane is a bulk mechanical search,
234
+ so reasoning on it is pure waste - and the model that could switch it off
235
+ publishes `none` and the four zero-retention models do not, so under the
236
+ ceiling that lane cannot ask for it. It runs on DeepSeek, which publishes
237
+ `low`, and that is the closest thing available.
238
+
239
+ **A model that publishes no effort level still spends what it spends.** It
240
+ runs at its natural rate. That is not a defect: it is the reason Big Pickle
241
+ is cheap to use, and the reason the kit asks for a tier only where the tier
242
+ is worth having.
106
243
 
107
244
  ## What the tier cannot do
108
245
 
@@ -147,6 +284,11 @@ mis-served by that:
147
284
  - `locate` asked for `low` effort and landed on a model that publishes no
148
285
  effort levels at all, so the request was dropped on the floor. It now lands
149
286
  on DeepSeek, which publishes `low`, and measures 3612 ms.
287
+ - `sweep` is a bulk mechanical search, so reasoning on it is pure waste. It
288
+ ran on LongCat at 420 reasoning tokens a message and 16.6 s - and no
289
+ zero-retention model publishes a `none` level, so the lane could not turn
290
+ reasoning off even when it asked to. It runs on DeepSeek now: 19 reasoning
291
+ tokens at 3.4 s.
150
292
  - `vision` landed on the same model, at 9-13 s, when MiniMax reads the same
151
293
  image in 7.7 s.
152
294
 
@@ -170,14 +312,22 @@ base profile would reveal.
170
312
 
171
313
  ### What the safe fleet gives up
172
314
 
173
- `sweep` is the one lane that loses its reason to exist under the ceiling.
315
+ `sweep` was the one lane that lost its reason to exist under the ceiling.
174
316
  Super VL is the only model on the tier that publishes a `none` effort level,
175
- so "reasoning off" is a job of its own; and Super VL is a may-train provider,
176
- so the ceiling excludes it. Under `fleet-safe` the lane falls back to
177
- LongCat, which publishes no effort levels at all, and it is a second `locate`
178
- with a 1M context window. The lane is kept, because `oc-harness lanes` should
179
- decide that from measurements rather than from a guess - but the docs say out
180
- loud that the capability is gone.
317
+ so "reasoning off" is a job of its own; and Super VL is a may-train
318
+ provider, so the ceiling excludes it. Under `fleet-safe` the lane ran on
319
+ LongCat, which publishes no effort levels at all, and it was a second
320
+ `locate` with a 1M context window.
321
+
322
+ It runs on DeepSeek now. Not LongCat: LongCat measured 420 reasoning tokens
323
+ a message at 16.6 seconds, which is the most expensive way to ask a model to
324
+ grep, and DeepSeek publishes a `low` level and measures 19 reasoning tokens
325
+ at 3.4. The lane is still not the unrestricted fleet's lane, and the
326
+ reasoning-off capability is still out of reach under a zero-retention
327
+ ceiling - but the cost of the substitute is no longer the whole story.
328
+
329
+ The lane is kept, because `oc-harness lanes` should decide that from
330
+ measurements rather than from a guess.
181
331
 
182
332
  ## Measuring, and cutting
183
333
 
@@ -187,19 +337,27 @@ is to count the requests:
187
337
  ```
188
338
  $ oc-harness lanes
189
339
 
190
- lane reqs avg ms model
191
- ----------------------------------------------------------------
192
- plan 7 2624 beam-free
193
- build 21 10104 minimax-m3.1-flash-free
194
- reviewer 2 4716 nemotron-3-ultra-free
340
+ lane reqs avg ms avg ctx model
341
+ ----------------------------------------------------------------------
342
+ plan 0 - - deepseek-v4.1-flash-free never used
343
+ build 87 9935 - minimax-m3.1-flash-free, deepseek-v4.1-flash-free
344
+ reviewer 0 - - glm-5.3-flash-free never used
195
345
  ...
196
- 8 lane(s) were never used cut candidates, not defects
346
+ 9 lane(s) were never used cut candidates, not defects
347
+ one request is one assistant message; avg ms covers the requests that finished
197
348
  ```
198
349
 
199
350
  A lane that is never used is a description the model reads on every turn and a
200
351
  free model doing nothing. Cut it from the profile's `roles`, and the lane
201
352
  leaves the config with it.
202
353
 
354
+ That is also the counter that says a lane is NOT earning its place, and it
355
+ is the honest one to act on: it counts the requests a working session
356
+ actually made. `node eval/models.mjs` answers the other question - how long
357
+ a lane would wait - and the two together are what a routing decision needs.
358
+ The counter alone would cut a lane that is slow but used, and the
359
+ measurement alone would keep a lane nobody calls.
360
+
203
361
  The counter reads `.space/telemetry.jsonl` in the project, written by the
204
362
  plugin: one record per assistant message, with the lane, the model and the
205
363
  duration. Two things about it are worth knowing, because both were wrong at
@@ -250,5 +408,7 @@ directory is exactly as dangerous as one in the plural one.
250
408
  config, so it is sized by the **narrowest** lane: 24,000 tokens for
251
409
  `fleet-safe`, whose narrowest lane has a 200k window, and 19,200 for
252
410
  `fleet-all`, whose narrowest lane is Big Pickle at a 160k input cap.
411
+ (Big Pickle's CATALOGUE context is 200k; the profile pins its input cap to
412
+ 160k, and the cap is the number that governs.)
253
413
  Sizing it by the widest model instead would reserve 125,829 tokens and
254
414
  leave every 200k lane with 74k usable.