opencode-harness-kit 0.4.3 → 0.4.5
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +569 -18
- package/CONTRIBUTING.md +249 -0
- package/GLOBAL.md +34 -5
- package/MIGRATION.md +151 -0
- package/README.md +354 -66
- package/SECURITY.md +30 -0
- package/bin/baseline.mjs +15 -7
- package/bin/git-hooks/pre-commit +41 -9
- package/bin/oc-harness.mjs +1268 -82
- package/dist/config.d.ts +127 -0
- package/dist/denylist.d.ts +33 -0
- package/dist/effort.d.ts +64 -0
- package/dist/guard.d.ts +23 -0
- package/dist/health.d.ts +43 -0
- package/dist/index.d.ts +9 -0
- package/dist/index.js +532 -149
- package/dist/plugin.d.ts +15 -0
- package/dist/profile.d.ts +399 -0
- package/dist/project.d.ts +8 -0
- package/dist/prune.d.ts +6 -0
- package/dist/router.d.ts +3 -0
- package/dist/rules.d.ts +109 -0
- package/dist/safe.d.ts +7 -0
- package/dist/telemetry-store.d.ts +12 -0
- package/dist/telemetry.d.ts +3 -0
- package/docs/EXIT-CODES.md +83 -0
- package/docs/KNOWN-LIMITS.md +45 -21
- package/docs/RESEARCH.md +20 -5
- package/docs/ROUTING.md +191 -31
- package/eval/RESULTS.md +457 -0
- package/eval/ab.mjs +743 -0
- package/eval/attribute.mjs +170 -0
- package/eval/flags.mjs +88 -0
- package/eval/measure-install.mjs +331 -0
- package/eval/measure-prune.mjs +339 -0
- package/eval/metrics.js +79 -0
- package/eval/models.mjs +302 -0
- package/eval/premises.mjs +200 -0
- package/eval/prepare.mjs +233 -0
- package/eval/run.mjs +376 -0
- package/eval/tasks-self.json +81 -0
- package/eval/tasks-ts.json +50 -0
- package/eval/tasks.json +83 -0
- package/eval/thresholds.mjs +134 -0
- package/eval/verify.mjs +287 -0
- package/package.json +44 -17
- package/profiles/fleet-all.json +6 -1
- package/profiles/fleet-safe.json +7 -3
- package/profiles/schema.json +182 -51
- package/prompts/build.md +4 -2
- package/skills/space-build/SKILL.md +4 -2
- package/skills/space-finish/SKILL.md +6 -2
- package/skills/space-plan/reference/task-template.md +13 -5
- package/src/config.js +182 -32
- package/src/denylist.js +95 -0
- package/src/driver/run.mjs +81 -16
- package/src/effort.js +15 -4
- package/src/guard.ts +165 -13
- package/src/health.js +80 -0
- package/src/index.ts +15 -4
- package/src/lane-usage.js +72 -18
- package/src/plugin-type-check.ts +38 -0
- package/src/plugin.ts +39 -0
- package/src/profile.js +225 -26
- package/src/project.js +47 -0
- package/src/prune.ts +110 -31
- package/src/router.ts +32 -63
- package/src/rules.js +99 -7
- package/src/safe.ts +27 -0
- package/src/sqlite.js +144 -0
- package/src/tasks.js +112 -29
- package/src/telemetry-store.ts +62 -0
- package/src/telemetry.ts +60 -57
- package/src/trust.js +70 -9
package/docs/KNOWN-LIMITS.md
CHANGED
|
@@ -96,8 +96,8 @@ a client cannot currently tell a short quota window from a permanent block.
|
|
|
96
96
|
|
|
97
97
|
## A copied plugin loads and does nothing
|
|
98
98
|
|
|
99
|
-
The bundle finds `profiles
|
|
100
|
-
|
|
99
|
+
The bundle finds `profiles/` and `lanes.json` by walking up from its own file,
|
|
100
|
+
and `retention.yml` next to them. A symlink resolves to the real path, so a
|
|
101
101
|
symlinked plugin finds the kit's data. A **copy** does not: it looks beside
|
|
102
102
|
itself, finds nothing, and every profile lookup fails.
|
|
103
103
|
|
|
@@ -284,23 +284,47 @@ unhealthy and failing over at the next start is the honest behaviour.
|
|
|
284
284
|
## Free models get withdrawn quickly
|
|
285
285
|
|
|
286
286
|
From one install's 20-day history, free model slots turned over in days:
|
|
287
|
-
|
|
288
|
-
|
|
289
|
-
|
|
290
|
-
|
|
291
|
-
|
|
292
|
-
|
|
293
|
-
|
|
294
|
-
|
|
295
|
-
|
|
296
|
-
|
|
297
|
-
|
|
298
|
-
|
|
299
|
-
|
|
300
|
-
|
|
301
|
-
|
|
302
|
-
|
|
303
|
-
|
|
287
|
+
`mimo-v2.5-free` lasted a day, `big-pickle` a day, `nemotron-3-ultra-free`
|
|
288
|
+
three days. Zen states free models are available "for a limited time" with no
|
|
289
|
+
date. Treat profiles as perishable.
|
|
290
|
+
|
|
291
|
+
The README's [Free models are perishable](../README.md#free-models-are-perishable-so-check-them)
|
|
292
|
+
section has the full list of withdrawn models, the four commands that check
|
|
293
|
+
for drift, and what `refresh` will and will not rewrite. The short version:
|
|
294
|
+
run `oc-harness doctor` before every release and whenever a lane starts
|
|
295
|
+
failing with a 429 or a withdrawal.
|
|
296
|
+
|
|
297
|
+
The tier turned over again since. Every one of this kit's original
|
|
298
|
+
profiles has been withdrawn — `space-bunny-free`, `x-preview-f-free`,
|
|
299
|
+
`deepseek-v4-flash-free`, `muse-spark-1.3-contributor-free` and
|
|
300
|
+
`mimo-v2.5-free` — and the default profile pointed at a model that no longer
|
|
301
|
+
existed.
|
|
302
|
+
|
|
303
|
+
## The install-wide measurement is an observation, and there is a controlled one
|
|
304
|
+
|
|
305
|
+
`npm run measure` compares the work before the kit landed with the work
|
|
306
|
+
after. It reported the context per request going **up** by 16% on one install,
|
|
307
|
+
with twice as many reads and twice as many messages in the second window.
|
|
308
|
+
|
|
309
|
+
That is a number about the jobs, not about the kit. The two windows are never
|
|
310
|
+
the same work, and "everything since the install" is a window that grows
|
|
311
|
+
without bound, so the comparison is a different one every time the script
|
|
312
|
+
runs.
|
|
313
|
+
|
|
314
|
+
`npm run measure` now says so: the windows are fixed and equal around the
|
|
315
|
+
split, the context drop is skipped with a note when the two sides are not
|
|
316
|
+
comparable, and the gate uses the number that IS comparable.
|
|
317
|
+
|
|
318
|
+
`eval/measure-prune.mjs` is that number. opencode stores the message parts
|
|
319
|
+
unpruned - the pruner rewrites the outgoing request, not the history - so the
|
|
320
|
+
stored parts of a real session are exactly what the pruner was given, and
|
|
321
|
+
replaying the real pruner over them measures exactly what it removed. On one
|
|
322
|
+
install: **80.5% of the context
|
|
323
|
+
across 60 real sessions**, 6,608 parts stubbed - the number
|
|
324
|
+
`npm run measure:prune` prints.
|
|
325
|
+
|
|
326
|
+
The controlled comparison of the two arms is `eval/ab.mjs`, and its result is
|
|
327
|
+
in `eval/RESULTS.md`.
|
|
304
328
|
|
|
305
329
|
## A model-driven compressor is not assumed to work
|
|
306
330
|
|
|
@@ -322,7 +346,7 @@ truncation directory, so re-reading is possible. But "re-run the command to see
|
|
|
322
346
|
it again" is more expensive than "scroll up". The stub is one line precisely so
|
|
323
347
|
an agent can decide whether it is worth it, and the stub says so.
|
|
324
348
|
|
|
325
|
-
A `
|
|
349
|
+
A `savedChars` of 0 in `prune.jsonl` is not a second, redundant pass. It means
|
|
326
350
|
`bytesAfter` did not come out below `bytesBefore` for that turn, which happens
|
|
327
351
|
when a stub is no shorter than the output it replaced (the stub's own label
|
|
328
352
|
costs more than a short body), or when a `reasoning` part was under the 200
|
|
@@ -346,7 +370,7 @@ compressor, or a session UI. Those exist and are good:
|
|
|
346
370
|
|---|---|
|
|
347
371
|
| model-driven compression | `@tarquinen/opencode-dcp` |
|
|
348
372
|
| cross-harness context proxy | `sleev.ai` |
|
|
349
|
-
| persistent memory | `opencode-
|
|
373
|
+
| persistent memory | [`opencode-mem`](https://github.com/tickernelz/opencode-mem), which `oc-harness use --memory` turns on |
|
|
350
374
|
| git worktrees | `kdcokenny/opencode-worktree` |
|
|
351
375
|
| memory in git markdown | `code-yeongyu/oh-my-opencode` (now `omo`) |
|
|
352
376
|
| agent DAGs | `spoons-and-mirrors/subtask2` |
|
package/docs/RESEARCH.md
CHANGED
|
@@ -47,13 +47,18 @@ ceiling that a normal session never reaches.
|
|
|
47
47
|
|
|
48
48
|
5,239 historical `bash` commands were replayed against the allowlist shapes:
|
|
49
49
|
|
|
50
|
-
| allowlist | prompts | share |
|
|
50
|
+
| allowlist | prompts | share of 5,239 |
|
|
51
51
|
|---|---|---|
|
|
52
52
|
| the narrow one in use | 4,994 | 95.3% |
|
|
53
53
|
| this kit's, first version | 1,204 | 23.0% |
|
|
54
|
-
| this kit's, current | 1,002 |
|
|
54
|
+
| this kit's, current | 1,002 | 19.1% |
|
|
55
55
|
|
|
56
|
-
The
|
|
56
|
+
The three rows share the denominator, and the arithmetic is theirs to be
|
|
57
|
+
checked: 4,994/5,239 is 95.3%, 1,204/5,239 is 23.0%, and 1,002/5,239 is
|
|
58
|
+
19.1%. The last figure was quoted as 18.5% in two places, which is a
|
|
59
|
+
different number for the same count.
|
|
60
|
+
|
|
61
|
+
The remaining 19.1% is the deliberately gated set, ranked by how often it
|
|
57
62
|
actually came up:
|
|
58
63
|
|
|
59
64
|
```
|
|
@@ -111,7 +116,7 @@ is what actually bounds context.
|
|
|
111
116
|
`bin/baseline.mjs` now prints all five numbers so this is re-measurable instead
|
|
112
117
|
of being re-argued.
|
|
113
118
|
|
|
114
|
-
**"Compaction never ran, therefore nothing is wrong."** Zero events across
|
|
119
|
+
**"Compaction never ran, therefore nothing is wrong."** Zero events across 153
|
|
115
120
|
sessions was read as "the setting is fine" rather than "nothing is being
|
|
116
121
|
bounded". Both readings were half right and the compound conclusion was wrong:
|
|
117
122
|
the configured `tail_turns` was inert *and* there was no bound on growth.
|
|
@@ -188,7 +193,14 @@ out.** Re-measured on 2026-10-20, lane by lane, over live calls: GLM is
|
|
|
188
193
|
4.4-5.8 seconds and not 16-27 (the old figure came from a much larger
|
|
189
194
|
prompt); MiMo reads an image in 19.2 seconds and not 8; and Nemotron Super VL
|
|
190
195
|
does **not** hang on a tool read, only on a file attached to the prompt, which
|
|
191
|
-
means the sweep lane is sound.
|
|
196
|
+
means the sweep lane is sound.
|
|
197
|
+
|
|
198
|
+
Those were the install's numbers, measured inside a working session. A
|
|
199
|
+
DIRECT measurement on 2026-10-21 - one line, the same prompt for every
|
|
200
|
+
model, from the process start - puts GLM at 7.4 seconds and Beam at 6.1, so
|
|
201
|
+
the two are not comparable and the routing page now says which is which.
|
|
202
|
+
The image figures are still the install's: a one-line prompt is not an
|
|
203
|
+
image. The measurement also showed the safe fleet
|
|
192
204
|
giving up more than it needed to: two lanes were on a model the ceiling had
|
|
193
205
|
forced them onto rather than the best one available for that job. The numbers,
|
|
194
206
|
and what changed because of them, are in [`ROUTING.md`](ROUTING.md).
|
|
@@ -283,3 +295,6 @@ and quietly replacing it.
|
|
|
283
295
|
session
|
|
284
296
|
- The live free tier, measured 2026-10-20: one request per lane on both
|
|
285
297
|
fleets, ten lanes on each, through real delegations
|
|
298
|
+
- The live free tier, measured 2026-10-21, directly: `node eval/models.mjs
|
|
299
|
+
--reps 5 --effort` - the same one-line prompt to every model the profiles
|
|
300
|
+
use, from the process start, with the `low` variant as well
|
package/docs/ROUTING.md
CHANGED
|
@@ -4,9 +4,26 @@ The fleet is the point of this kit. One session, ten lanes, ten models, and a
|
|
|
4
4
|
human who never switches a model by hand.
|
|
5
5
|
|
|
6
6
|
This page records what the lanes are, which model serves each one, and the
|
|
7
|
-
measurements behind those choices.
|
|
8
|
-
|
|
9
|
-
|
|
7
|
+
measurements behind those choices.
|
|
8
|
+
|
|
9
|
+
There are **two** sets of numbers here, and they measure different things:
|
|
10
|
+
|
|
11
|
+
- The **install tables** are the telemetry of a real install: `oc-harness
|
|
12
|
+
lanes` over the requests a working session actually made, on
|
|
13
|
+
**2026-10-20**. They are the wait a person really had, and the `build` row
|
|
14
|
+
is not comparable with the others, because it counts the turns that did the
|
|
15
|
+
delegating.
|
|
16
|
+
- The **direct table** is `node eval/models.mjs` - one line, one model, five
|
|
17
|
+
requests, whole-process wall time, on **2026-10-21**. It is the same
|
|
18
|
+
prompt for every model, so it compares models with each other, and it
|
|
19
|
+
includes the process start, which the install tables do not.
|
|
20
|
+
|
|
21
|
+
Neither is the answer on its own. The install tables say what happened; the
|
|
22
|
+
direct table says what a lane would wait for if the lane were all there was.
|
|
23
|
+
|
|
24
|
+
Both are reproducible: `oc-harness doctor` for the catalogue, `oc-harness
|
|
25
|
+
lanes` for the install, and `node eval/models.mjs --reps 5 --effort` for the
|
|
26
|
+
direct table.
|
|
10
27
|
|
|
11
28
|
## The lanes
|
|
12
29
|
|
|
@@ -23,7 +40,7 @@ models. A profile says which lanes it enables and which model serves each one.
|
|
|
23
40
|
| `vision` | read an image or a screenshot | mimo-v2.6-flash-free | minimax-m3.1-flash-free |
|
|
24
41
|
| `doc` | read a long document or a spec | glm-5.3-flash-free | glm-5.3-flash-free |
|
|
25
42
|
| `schema` | produce JSON or a JSON schema | deepseek-v4.1-flash-free | deepseek-v4.1-flash-free |
|
|
26
|
-
| `sweep` | sweep a lot of code mechanically | nemotron-3.5-super-vl-free |
|
|
43
|
+
| `sweep` | sweep a lot of code mechanically | nemotron-3.5-super-vl-free | deepseek-v4.1-flash-free |
|
|
27
44
|
| `longform` | read a megabyte, or write alternatives | longcat-2.5-preview-free | longcat-2.5-preview-free |
|
|
28
45
|
|
|
29
46
|
`fleet-all` gives every lane the model built for it: ten jobs, ten models,
|
|
@@ -35,6 +52,17 @@ ceiling, and it is worth paying only when the model that takes over is the
|
|
|
35
52
|
best one available *for that job*, which is why the two fleets order their
|
|
36
53
|
chains differently (see below).
|
|
37
54
|
|
|
55
|
+
The `sweep` lane is the one that was worth changing rather than accepting.
|
|
56
|
+
A bulk mechanical search is the one job where reasoning is pure waste, and
|
|
57
|
+
the model it ran on reasoned 420 tokens a message at 16.6 seconds - the
|
|
58
|
+
most expensive way to ask a model to grep. It runs on DeepSeek now, which is
|
|
59
|
+
zero-retention, publishes a `low` level, and measures 19 reasoning tokens at
|
|
60
|
+
3.4 seconds.
|
|
61
|
+
|
|
62
|
+
Under that ceiling every lane that asks for a tier GETS one. The safe fleet
|
|
63
|
+
has no lane with a dropped tier request; the unrestricted fleet has three, and
|
|
64
|
+
the measurement says they are the three cheapest reasoners on the tier.
|
|
65
|
+
|
|
38
66
|
## Measured, per lane
|
|
39
67
|
|
|
40
68
|
Every number here is a live call, not a catalog reading. A **request** is one
|
|
@@ -77,11 +105,88 @@ that did the delegating, so they count the work of the other nine lanes. A
|
|
|
77
105
|
one-line `build` reply on its own measured **3130 ms** and **21137 ms** on a
|
|
78
106
|
cold start.
|
|
79
107
|
|
|
108
|
+
## Measured directly, one line, one model
|
|
109
|
+
|
|
110
|
+
node eval/models.mjs --reps 5 --effort
|
|
111
|
+
|
|
112
|
+
Every model gets the same prompt - one line, one word back - and the same
|
|
113
|
+
five requests. `wall` is the whole `opencode run`, so it includes the
|
|
114
|
+
process start and the plugin load. `reasoning` is the tokens the model
|
|
115
|
+
spent thinking, which on a prompt this small is close to nothing for most of
|
|
116
|
+
them: the difference between a tier and no tier shows up on real work, not
|
|
117
|
+
on a one-line reply.
|
|
118
|
+
|
|
119
|
+
| model | wall avg | median | max | reasoning | wall @low | reasoning @low |
|
|
120
|
+
|---|---|---|---|---|---|---|
|
|
121
|
+
| `beam-free` | 7,800 | **6,118** | 13,782 | 24 | 6,308 | 28 |
|
|
122
|
+
| `minimax-m3.1-flash-free` | 7,114 | 6,903 | 8,448 | 0 | **6,292** | 0 |
|
|
123
|
+
| `nemotron-3.5-lightning-free` | 8,140 | 6,882 | 13,554 | 86 | — | — |
|
|
124
|
+
| `glm-5.3-flash-free` | 10,604 | 7,408 | 21,002 | 0 | 10,718 | 12 |
|
|
125
|
+
| `nemotron-3-ultra-free` | 9,282 | 7,847 | 15,402 | 36 | — | — |
|
|
126
|
+
| `nemotron-3.5-super-vl-free` | 10,404 | 8,353 | 19,763 | 15 | 10,988 | 17 |
|
|
127
|
+
| `mimo-v2.6-flash-free` | 8,875 | 9,345 | 9,635 | 8 | — | — |
|
|
128
|
+
| `big-pickle` | 9,983 | 10,438 | 11,372 | 8 | — | — |
|
|
129
|
+
| `deepseek-v4.1-flash-free` | 15,840 | 10,659 | 25,805 | 0 | 14,193 | 46 |
|
|
130
|
+
| `longcat-2.5-preview-free` | 13,099 | 11,996 | 16,991 | 16 | — | — |
|
|
131
|
+
|
|
132
|
+
**What this says that the install tables do not.** The margin between the
|
|
133
|
+
fastest model and the fourth is under a second - 6.1 s for Beam against 7.4 s
|
|
134
|
+
for GLM - so a plan that has to be quick does not need the fastest model, it
|
|
135
|
+
needs any of the first four. The spread at the other end is real: LongCat
|
|
136
|
+
costs twice what Beam costs, on the same one-line question.
|
|
137
|
+
|
|
138
|
+
**And the `low` column is not a reason to ask for `low` on a small job.**
|
|
139
|
+
On a prompt this small the difference is inside the noise, and on one of them
|
|
140
|
+
it is the wrong way round. The tier earns its keep on real work, and that
|
|
141
|
+
is the measurement below, on the install's own messages.
|
|
142
|
+
|
|
143
|
+
A dash in the `@low` columns means the model publishes **no `low` level**,
|
|
144
|
+
so there is nothing to ask for: it runs at its natural rate. The script
|
|
145
|
+
reads the catalogue to know which is which, rather than asking every model
|
|
146
|
+
and reporting an answer it did not use.
|
|
147
|
+
|
|
148
|
+
The model ids are shown without their provider; the script prints
|
|
149
|
+
`opencode/beam-free`, and either spelling works with `--model`.
|
|
150
|
+
|
|
151
|
+
**The free tier is noisy, and the table says so.** Five requests is five
|
|
152
|
+
requests: `deepseek` had a 25.8 s outlier in a 10.7 s median, and Beam had a
|
|
153
|
+
13.8 s one in a 6.1 s median. The median is the number to read, and a
|
|
154
|
+
single request is not a measurement of anything.
|
|
155
|
+
|
|
80
156
|
**What the safe ceiling costs, measured rather than assumed.** Planning is the
|
|
81
|
-
lane that pays: 2624 ms on Beam against 5106 ms on
|
|
82
|
-
|
|
83
|
-
|
|
84
|
-
|
|
157
|
+
lane that pays: the install measured 2624 ms on Beam against 5106 ms on
|
|
158
|
+
DeepSeek, and the direct measurement says the same thing a different way -
|
|
159
|
+
6.1 s against 10.7 s, about three quarters more.
|
|
160
|
+
|
|
161
|
+
Reviewing does not pay at all. The install measured 4716 ms on Nemotron 3
|
|
162
|
+
Ultra against 5757 ms on GLM, and the direct measurement has them level, with
|
|
163
|
+
GLM marginally ahead: 7.4 s against 7.8 s. Either way, the model built for
|
|
164
|
+
reviewing is not faster than the one the ceiling puts in its place, and
|
|
165
|
+
neither is slow.
|
|
166
|
+
|
|
167
|
+
Locating and verifying get *faster*, because DeepSeek publishes a `low` effort
|
|
168
|
+
level and Big Pickle publishes none at all.
|
|
169
|
+
|
|
170
|
+
**What a tier is worth, measured on this install rather than assumed.** The
|
|
171
|
+
effort level is the cheapest lever there is: DeepSeek measures 320 reasoning
|
|
172
|
+
tokens a message at `max` and 19 at `low`, a factor of 17, and a lane that
|
|
173
|
+
asks for a tier and lands on a model that publishes none runs at that model's
|
|
174
|
+
NATURAL rate instead. So the tier request is worth protecting.
|
|
175
|
+
|
|
176
|
+
The three lanes in `fleet-all` whose model cannot serve the tier - `reviewer`
|
|
177
|
+
on Nemotron 3 Ultra, `locate` on Nemotron 3.5 Lightning, `verify` on Big
|
|
178
|
+
Pickle - are the fleet's CHEAPEST reasoners: 116, 59 and 18 reasoning tokens a
|
|
179
|
+
message, against 289 for `build` and 198 for `plan`, which keep their tier on
|
|
180
|
+
purpose. The tier is the lever, and it is not where the tokens are. The
|
|
181
|
+
config header says so where it happens, in the form `locate=unset(asked low)`.
|
|
182
|
+
|
|
183
|
+
One lane was not cheap, and it was fixed. `sweep` is a bulk mechanical
|
|
184
|
+
search, so reasoning on it is pure waste, and it ran on LongCat: 420 reasoning
|
|
185
|
+
tokens a message at 16.6 s, the most expensive way to ask a model to grep. No
|
|
186
|
+
zero-retention model publishes a `none` level, so the lane could not even turn
|
|
187
|
+
reasoning off when it asked to. It runs on DeepSeek now, which is
|
|
188
|
+
zero-retention, publishes a `low` level, and measures 19 reasoning tokens at
|
|
189
|
+
3.4 s.
|
|
85
190
|
|
|
86
191
|
And one lane gets faster rather than slower. MiMo reads an image in 19.2 s;
|
|
87
192
|
MiniMax reads the same image in 7.7 s. The unrestricted fleet put `vision` on
|
|
@@ -93,16 +198,48 @@ choice anyway.
|
|
|
93
198
|
|
|
94
199
|
| model | measured | why it owns that lane |
|
|
95
200
|
|---|---|---|
|
|
96
|
-
| minimax-m3.1-flash-free |
|
|
97
|
-
| beam-free | **
|
|
98
|
-
| nemotron-3-ultra-free |
|
|
99
|
-
| nemotron-3.5-lightning-free |
|
|
100
|
-
| big-pickle |
|
|
101
|
-
| mimo-v2.6-flash-free |
|
|
102
|
-
| glm-5.3-flash-free |
|
|
103
|
-
| deepseek-v4.1-flash-free |
|
|
104
|
-
| nemotron-3.5-super-vl-free | 4
|
|
105
|
-
| longcat-2.5-preview-free |
|
|
201
|
+
| minimax-m3.1-flash-free | 6.9 s median for a one-line reply, 512k output cap - the widest on the tier, reads an image in 7.7 s | The widest output cap on the tier and a stable tool call, so the lane that writes code fits, and it is also the fastest measured image reader. |
|
|
202
|
+
| beam-free | **6.1 s median, the fastest of the ten**, 1M context, structured output | Planning is a reading-and-reasoning job with a long context and a structured answer. It is the fastest measured, but only by under a second over the next three - so the reason to pick it is the 1M context and the structured output as much as the speed. |
|
|
203
|
+
| nemotron-3-ultra-free | 7.8 s median, but **34 s on a cold first request** | The largest open MoE available, which is what a two-pass review needs. The cold start is worth knowing about before blaming the route. |
|
|
204
|
+
| nemotron-3.5-lightning-free | 6.9 s median, 13.6 s at worst, no selectable effort | No effort level, so it spends nothing on reasoning, and it can emit a file larger than 128k tokens. |
|
|
205
|
+
| big-pickle | 10.4 s median, structured output, no effort levels, no images | Stable, and it answers in a shape a program can read. |
|
|
206
|
+
| mimo-v2.6-flash-free | 9.3 s median for a line of text, **19.2 s average with an image** | It reads images correctly, which is the job. It publishes audio input too, and that path is unreachable (see below). |
|
|
207
|
+
| glm-5.3-flash-free | 7.4 s median, structured output, 3 effort levels | Careful and quick enough. A previous measurement put it at 16-27 s on a larger prompt; on the work a lane actually does it is about seven seconds. |
|
|
208
|
+
| deepseek-v4.1-flash-free | 10.7 s median, structured output, reasoning + agentic coding | The natural lane for JSON a program has to parse, and the best all-rounder inside a zero-retention ceiling. It was the slowest of the ten on the one-line measurement - the tier is free, and free is not fast. |
|
|
209
|
+
| nemotron-3.5-super-vl-free | 8.4 s median, **the only model publishing a `none` effort level** | A lane that turns reasoning off entirely is a job of its own, and no other model can do it. |
|
|
210
|
+
| longcat-2.5-preview-free | 12.0 s median - the slowest of the ten, 1M context, reads images, temperature on | The long-context lane. A preview endpoint: it can be withdrawn without a rename. |
|
|
211
|
+
|
|
212
|
+
## What each model publishes
|
|
213
|
+
|
|
214
|
+
From the live catalogue, 2026-10-21. `oc-harness doctor` re-checks all of
|
|
215
|
+
it, and a model that is withdrawn says so rather than being measured.
|
|
216
|
+
|
|
217
|
+
| model | effort levels | context | output cap | reads |
|
|
218
|
+
|---|---|---|---|---|
|
|
219
|
+
| `beam-free` | low, medium, high, xhigh, max | 1M | 131k | text |
|
|
220
|
+
| `big-pickle` | **none** | 200k | 32k | text |
|
|
221
|
+
| `deepseek-v4.1-flash-free` | low, high, max | 200k | 32k | text, image |
|
|
222
|
+
| `minimax-m3.1-flash-free` | low, medium, high, xhigh, max | 1M | **512k** | text, image, video |
|
|
223
|
+
| `nemotron-3-ultra-free` | **none** | 1M | 128k | text |
|
|
224
|
+
| `nemotron-3.5-lightning-free` | **none** | 262k | **262k** | text |
|
|
225
|
+
| `mimo-v2.6-flash-free` | **none** | 200k | 32k | text, image, audio, video |
|
|
226
|
+
| `glm-5.3-flash-free` | low, high, max | 200k | 32k | text, image, video, pdf |
|
|
227
|
+
| `nemotron-3.5-super-vl-free` | **none**, minimal, low, medium, high, xhigh | 1M | 131k | text, image |
|
|
228
|
+
| `longcat-2.5-preview-free` | **none** | 1M | 131k | text, image |
|
|
229
|
+
|
|
230
|
+
Two things fall out of this table that the prose above rests on.
|
|
231
|
+
|
|
232
|
+
**Only one model can turn reasoning off entirely**, and it is
|
|
233
|
+
`nemotron-3.5-super-vl-free`. The `sweep` lane is a bulk mechanical search,
|
|
234
|
+
so reasoning on it is pure waste - and the model that could switch it off
|
|
235
|
+
publishes `none` and the four zero-retention models do not, so under the
|
|
236
|
+
ceiling that lane cannot ask for it. It runs on DeepSeek, which publishes
|
|
237
|
+
`low`, and that is the closest thing available.
|
|
238
|
+
|
|
239
|
+
**A model that publishes no effort level still spends what it spends.** It
|
|
240
|
+
runs at its natural rate. That is not a defect: it is the reason Big Pickle
|
|
241
|
+
is cheap to use, and the reason the kit asks for a tier only where the tier
|
|
242
|
+
is worth having.
|
|
106
243
|
|
|
107
244
|
## What the tier cannot do
|
|
108
245
|
|
|
@@ -147,6 +284,11 @@ mis-served by that:
|
|
|
147
284
|
- `locate` asked for `low` effort and landed on a model that publishes no
|
|
148
285
|
effort levels at all, so the request was dropped on the floor. It now lands
|
|
149
286
|
on DeepSeek, which publishes `low`, and measures 3612 ms.
|
|
287
|
+
- `sweep` is a bulk mechanical search, so reasoning on it is pure waste. It
|
|
288
|
+
ran on LongCat at 420 reasoning tokens a message and 16.6 s - and no
|
|
289
|
+
zero-retention model publishes a `none` level, so the lane could not turn
|
|
290
|
+
reasoning off even when it asked to. It runs on DeepSeek now: 19 reasoning
|
|
291
|
+
tokens at 3.4 s.
|
|
150
292
|
- `vision` landed on the same model, at 9-13 s, when MiniMax reads the same
|
|
151
293
|
image in 7.7 s.
|
|
152
294
|
|
|
@@ -170,14 +312,22 @@ base profile would reveal.
|
|
|
170
312
|
|
|
171
313
|
### What the safe fleet gives up
|
|
172
314
|
|
|
173
|
-
`sweep`
|
|
315
|
+
`sweep` was the one lane that lost its reason to exist under the ceiling.
|
|
174
316
|
Super VL is the only model on the tier that publishes a `none` effort level,
|
|
175
|
-
so "reasoning off" is a job of its own; and Super VL is a may-train
|
|
176
|
-
so the ceiling excludes it. Under `fleet-safe` the lane
|
|
177
|
-
LongCat, which publishes no effort levels at all, and it
|
|
178
|
-
with a 1M context window.
|
|
179
|
-
|
|
180
|
-
|
|
317
|
+
so "reasoning off" is a job of its own; and Super VL is a may-train
|
|
318
|
+
provider, so the ceiling excludes it. Under `fleet-safe` the lane ran on
|
|
319
|
+
LongCat, which publishes no effort levels at all, and it was a second
|
|
320
|
+
`locate` with a 1M context window.
|
|
321
|
+
|
|
322
|
+
It runs on DeepSeek now. Not LongCat: LongCat measured 420 reasoning tokens
|
|
323
|
+
a message at 16.6 seconds, which is the most expensive way to ask a model to
|
|
324
|
+
grep, and DeepSeek publishes a `low` level and measures 19 reasoning tokens
|
|
325
|
+
at 3.4. The lane is still not the unrestricted fleet's lane, and the
|
|
326
|
+
reasoning-off capability is still out of reach under a zero-retention
|
|
327
|
+
ceiling - but the cost of the substitute is no longer the whole story.
|
|
328
|
+
|
|
329
|
+
The lane is kept, because `oc-harness lanes` should decide that from
|
|
330
|
+
measurements rather than from a guess.
|
|
181
331
|
|
|
182
332
|
## Measuring, and cutting
|
|
183
333
|
|
|
@@ -187,19 +337,27 @@ is to count the requests:
|
|
|
187
337
|
```
|
|
188
338
|
$ oc-harness lanes
|
|
189
339
|
|
|
190
|
-
lane
|
|
191
|
-
|
|
192
|
-
plan
|
|
193
|
-
build
|
|
194
|
-
reviewer
|
|
340
|
+
lane reqs avg ms avg ctx model
|
|
341
|
+
----------------------------------------------------------------------
|
|
342
|
+
plan 0 - - deepseek-v4.1-flash-free never used
|
|
343
|
+
build 87 9935 - minimax-m3.1-flash-free, deepseek-v4.1-flash-free
|
|
344
|
+
reviewer 0 - - glm-5.3-flash-free never used
|
|
195
345
|
...
|
|
196
|
-
|
|
346
|
+
9 lane(s) were never used cut candidates, not defects
|
|
347
|
+
one request is one assistant message; avg ms covers the requests that finished
|
|
197
348
|
```
|
|
198
349
|
|
|
199
350
|
A lane that is never used is a description the model reads on every turn and a
|
|
200
351
|
free model doing nothing. Cut it from the profile's `roles`, and the lane
|
|
201
352
|
leaves the config with it.
|
|
202
353
|
|
|
354
|
+
That is also the counter that says a lane is NOT earning its place, and it
|
|
355
|
+
is the honest one to act on: it counts the requests a working session
|
|
356
|
+
actually made. `node eval/models.mjs` answers the other question - how long
|
|
357
|
+
a lane would wait - and the two together are what a routing decision needs.
|
|
358
|
+
The counter alone would cut a lane that is slow but used, and the
|
|
359
|
+
measurement alone would keep a lane nobody calls.
|
|
360
|
+
|
|
203
361
|
The counter reads `.space/telemetry.jsonl` in the project, written by the
|
|
204
362
|
plugin: one record per assistant message, with the lane, the model and the
|
|
205
363
|
duration. Two things about it are worth knowing, because both were wrong at
|
|
@@ -250,5 +408,7 @@ directory is exactly as dangerous as one in the plural one.
|
|
|
250
408
|
config, so it is sized by the **narrowest** lane: 24,000 tokens for
|
|
251
409
|
`fleet-safe`, whose narrowest lane has a 200k window, and 19,200 for
|
|
252
410
|
`fleet-all`, whose narrowest lane is Big Pickle at a 160k input cap.
|
|
411
|
+
(Big Pickle's CATALOGUE context is 200k; the profile pins its input cap to
|
|
412
|
+
160k, and the cap is the number that governs.)
|
|
253
413
|
Sizing it by the widest model instead would reserve 125,829 tokens and
|
|
254
414
|
leave every 200k lane with 74k usable.
|