opencode-harness-kit 0.5.7 → 0.6.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +330 -0
- package/README.md +241 -11
- package/bin/baseline.mjs +16 -1
- package/bin/oc-harness.mjs +528 -42
- package/dist/index.js +294 -57
- package/dist/profile.d.ts +2 -1
- package/dist/rules.d.ts +6 -0
- package/docs/KNOWN-LIMITS.md +22 -2
- package/lanes.json +2 -2
- package/package.json +1 -1
- package/profiles/beam-free.json +2 -3
- package/profiles/deepseek-v4.1-flash-free.json +1 -2
- package/profiles/fleet-all.json +6 -14
- package/profiles/fleet-safe.json +8 -19
- package/profiles/glm-5.3-flash-free.json +1 -2
- package/profiles/index.json +4 -11
- package/profiles/mimo-v2.6-flash-free.json +1 -2
- package/profiles/minimax-m3.1-flash-free.json +1 -2
- package/profiles/nemotron-3.5-lightning-free.json +2 -3
- package/profiles/nemotron-3.5-super-vl-free.json +1 -2
- package/profiles/schema.json +8 -0
- package/retention.yml +0 -9
- package/src/config.js +9 -1
- package/src/profile.js +14 -1
- package/src/prune.ts +90 -5
- package/src/research.js +462 -0
- package/src/rules.js +303 -21
- package/profiles/longcat-2.5-preview-free.json +0 -45
package/CHANGELOG.md
CHANGED
|
@@ -12,6 +12,336 @@ is not happening.
|
|
|
12
12
|
|
|
13
13
|
---
|
|
14
14
|
|
|
15
|
+
## [Unreleased]
|
|
16
|
+
|
|
17
|
+
## [0.6.0] — 2026-10-22
|
|
18
|
+
|
|
19
|
+
Published. The tag is what cut it: `git tag v0.6.0 && git push origin
|
|
20
|
+
main v0.6.0`, and the publish job watches for exactly that.
|
|
21
|
+
|
|
22
|
+
A minor: a command for the half of the perishable-tier question nobody was
|
|
23
|
+
answering, and the fix that made it answerable.
|
|
24
|
+
|
|
25
|
+
The free tier turns over in days. `doctor` said whether the model you depend
|
|
26
|
+
on was still there; it said nothing about what had come to replace it, so a
|
|
27
|
+
withdrawal left a hole only a person could fill by hand out of a catalogue
|
|
28
|
+
dump. `research` fills the hole with a shortlist, and `refresh` takes the
|
|
29
|
+
dead rung out of the chain it was in.
|
|
30
|
+
|
|
31
|
+
Two of the bugs here were the same bug wearing different clothes: a check
|
|
32
|
+
that asked "is this model still IN the catalogue" and called the answer
|
|
33
|
+
"yes, it can run". models.dev does not delete a model the provider pulls - it
|
|
34
|
+
keeps the record and marks it `status: "deprecated"`. So this kit's own
|
|
35
|
+
default fleet was carrying a dead fallback rung in **all ten lanes** while
|
|
36
|
+
`doctor` called it healthy.
|
|
37
|
+
|
|
38
|
+
### Added
|
|
39
|
+
|
|
40
|
+
- **`refresh` now takes the dead rung out.** `research` tells you a rung is
|
|
41
|
+
dead; `refresh` removes it. A chain names the models that could serve a
|
|
42
|
+
lane, best first, and a model the provider has withdrawn cannot run - so it
|
|
43
|
+
is dead weight in a chain whose entire job is to survive a withdrawal.
|
|
44
|
+
|
|
45
|
+
A lane whose own primary was withdrawn takes the next live rung of its own
|
|
46
|
+
chain, because a chain has to start with its model. The fallback queue
|
|
47
|
+
loses its dead entries, and the utility route moves to a model the profile
|
|
48
|
+
still reaches.
|
|
49
|
+
|
|
50
|
+
**The refresh only ever REMOVES.** Putting a model into a lane changes what
|
|
51
|
+
that lane does, and that is a person's decision - it is what `research` is
|
|
52
|
+
for. A script that quietly added a candidate would be putting a model whose
|
|
53
|
+
terms nobody has read into the path of every prompt.
|
|
54
|
+
|
|
55
|
+
Two cases a script cannot repair are reported rather than papered over: a
|
|
56
|
+
profile whose own model was withdrawn (there is no chain behind it to fall
|
|
57
|
+
back to, so the honest fix is to retire the profile, and that is a person's
|
|
58
|
+
call), and a lane left with no model at all.
|
|
59
|
+
|
|
60
|
+
- **`oc-harness research` — the half of the perishable-tier question nobody
|
|
61
|
+
was answering.** `doctor` says whether the model you depend on is still
|
|
62
|
+
there. It said nothing about what has come to replace it, so a withdrawal
|
|
63
|
+
left a hole only a person could fill, by hand, out of a catalogue dump.
|
|
64
|
+
|
|
65
|
+
The new command reads the live catalogue and reports the models in use that
|
|
66
|
+
have been withdrawn — split into the ones that are a profile's own base
|
|
67
|
+
model and the ones that are a rung in a lane chain — then the lanes of the
|
|
68
|
+
default profile with a count of the rungs still alive in each, then the
|
|
69
|
+
candidates: live, free models the kit does not use yet, each with its
|
|
70
|
+
context window, its output cap, the effort levels it publishes, and **which
|
|
71
|
+
lanes it could serve**.
|
|
72
|
+
|
|
73
|
+
"Which lanes it could serve" is the part that makes a candidate usable
|
|
74
|
+
rather than merely interesting. Every lane needs tool calls, the `vision`
|
|
75
|
+
lane needs a model that takes an image, and a lane whose role asks for
|
|
76
|
+
`max` needs a model that publishes `max`. A model that publishes no effort
|
|
77
|
+
levels at all counts as a match, because there is nothing to mismatch.
|
|
78
|
+
|
|
79
|
+
The logic lives in one file, [`src/research.js`](src/research.js), with no
|
|
80
|
+
I/O in it, and the command is a thin layer over it.
|
|
81
|
+
|
|
82
|
+
**It writes nothing** — not `retention.yml`, not a profile, not the config.
|
|
83
|
+
That is the line `refresh` draws, for the same reason: `retention.yml` is a
|
|
84
|
+
human judgement about a provider's terms, and no script can read a
|
|
85
|
+
provider's terms and come back with a privacy verdict. A candidate with no
|
|
86
|
+
verdict is reported as `retention: unknown` and carries an explicit blocker
|
|
87
|
+
rather than being quietly filed under whichever tier it is cheapest on. The
|
|
88
|
+
output is a shortlist to argue with, not a decision to accept.
|
|
89
|
+
|
|
90
|
+
`research --json` prints one parseable object and nothing else, so a
|
|
91
|
+
scheduler can read it.
|
|
92
|
+
|
|
93
|
+
### Fixed
|
|
94
|
+
|
|
95
|
+
- **`doctor` reported a withdrawn model as a healthy fallback.** The chain
|
|
96
|
+
check asked whether the model was still *in* the catalogue. models.dev does
|
|
97
|
+
not delete a model the provider pulls: it keeps the record and marks it
|
|
98
|
+
`status: "deprecated"`. So "is it still there" and "can it still run" are
|
|
99
|
+
two different questions, and the kit's own default fleet carried a dead
|
|
100
|
+
fallback rung in **all ten lanes** while `doctor` called it healthy. The
|
|
101
|
+
message it printed already said "withdrawn" — only the filter was wrong,
|
|
102
|
+
which is exactly why the dead rung stayed invisible.
|
|
103
|
+
|
|
104
|
+
- **The pruner's ceiling on the distinct outputs did not bound what it kept.**
|
|
105
|
+
The rule keeps the newest copy of each distinct output, and the ceiling on
|
|
106
|
+
that is what stops it being a leak. But the ceiling charged the budget for
|
|
107
|
+
the *tool* parts only, while the rule keeps the *whole message*, verbatim,
|
|
108
|
+
every part in it. A message with a 300-character tool output and a
|
|
109
|
+
4,000-character reasoning part therefore cost the budget 300 and kept
|
|
110
|
+
4,300, and twenty such messages kept 40,000 characters of reasoning under a
|
|
111
|
+
5,000-character ceiling — eight times the budget, carrying exactly the
|
|
112
|
+
reasoning the pruner exists to stub. The ceiling now charges what the rule
|
|
113
|
+
actually keeps.
|
|
114
|
+
|
|
115
|
+
- **A profile that could not be loaded turned the pruner off without a word.**
|
|
116
|
+
The lookup sat in an empty `catch`, so a marker naming a renamed or missing
|
|
117
|
+
profile left the pruner disabled with no line in the log, nothing on the
|
|
118
|
+
console, and nothing to show the user why their context kept growing. A
|
|
119
|
+
project in that state could not tell the difference between "the pruner has
|
|
120
|
+
nothing to do" and "the pruner is broken". The failure is now reported once
|
|
121
|
+
per session, by name.
|
|
122
|
+
|
|
123
|
+
### Removed
|
|
124
|
+
|
|
125
|
+
- **The `longcat-2.5-preview-free` profile.** The provider withdrew the
|
|
126
|
+
model. A withdrawn model is not a profile that still works - it is a route
|
|
127
|
+
that fails on contact - and this one was the *only* model its profile named,
|
|
128
|
+
so there was no chain behind it to fall back to. The profile is retired and
|
|
129
|
+
recorded in `profiles/index.json` under `withdrawn`, with its retention
|
|
130
|
+
verdict removed from `retention.yml` along with it: a verdict about a model
|
|
131
|
+
that cannot run is not a verdict about anything.
|
|
132
|
+
|
|
133
|
+
The two fleets are unaffected in kind: `longcat` was a fallback rung in
|
|
134
|
+
every lane, and each of those lanes now runs on the rung above it. The
|
|
135
|
+
default fleet loses its third rung, so it now runs its ten lanes on the
|
|
136
|
+
three zero-retention models still live on the tier.
|
|
137
|
+
|
|
138
|
+
- **The demo video is not published with the repository.** The newer take —
|
|
139
|
+
thirty seconds, made on a machine with no screen, the terminal in it the
|
|
140
|
+
real output of the real commands — was under `docs/demo/` for the 0.5.8
|
|
141
|
+
release, and it is out of the repository now.
|
|
142
|
+
|
|
143
|
+
A take is about one machine and one moment, and so is the recording of it.
|
|
144
|
+
The `.gitignore` already said as much: "a recording stays on the machine
|
|
145
|
+
that made it until somebody decides, deliberately, to publish it". The
|
|
146
|
+
decision went the other way, so the file went back where that rule keeps
|
|
147
|
+
it — `video/build/`, which is ignored — and the README no longer links a
|
|
148
|
+
video the repository does not carry.
|
|
149
|
+
|
|
150
|
+
What is published is still the old `docs/demo/pitch.mp4`: four minutes
|
|
151
|
+
forty-seven seconds, recorded live from a Mac terminal, with a voice. The
|
|
152
|
+
npm package never carried a video at all.
|
|
153
|
+
|
|
154
|
+
## [0.5.8] — 2026-10-22
|
|
155
|
+
|
|
156
|
+
Published. The tag is what cut it: `git tag v0.5.8 && git push origin
|
|
157
|
+
main v0.5.8`, and the publish job watches for exactly that.
|
|
158
|
+
|
|
159
|
+
A minor: one new command, and eleven silent failures closed.
|
|
160
|
+
|
|
161
|
+
### Added
|
|
162
|
+
|
|
163
|
+
- **`oc-harness auto` — the whole reconcile in one command, asking nothing.**
|
|
164
|
+
`refresh`, `install`, `use` and `verify` are each a step somebody has to
|
|
165
|
+
remember, in the right order, with the right flags; an agent that has to
|
|
166
|
+
keep an install working has to know all of that, and has to know which of
|
|
167
|
+
them are allowed to destroy something.
|
|
168
|
+
|
|
169
|
+
The profile is the one the installed config names, or the default. A file
|
|
170
|
+
of yours that the installer would overwrite is **left alone and
|
|
171
|
+
reported**, not overwritten and not asked about — the answer a person
|
|
172
|
+
would give. `--offline` skips the refresh, for a machine that cannot
|
|
173
|
+
reach models.dev. `--json` prints a report a script can read, and nothing
|
|
174
|
+
else on stdout.
|
|
175
|
+
|
|
176
|
+
`--yes` still means consent: a profile whose models may train on your
|
|
177
|
+
prompts is refused without it, exactly as `use` refuses it. The automation
|
|
178
|
+
is about the mechanics, not about the consent.
|
|
179
|
+
|
|
180
|
+
`init` is unchanged. It is the front door for a person, and it still says
|
|
181
|
+
so when there is no terminal to ask on.
|
|
182
|
+
|
|
183
|
+
- **A demo whose numbers are current.** `docs/demo/audit.mp4` — thirty
|
|
184
|
+
seconds, made on a machine with no screen, no camera and no ffmpeg: the
|
|
185
|
+
scene is drawn by a browser, the terminal in it is the real output of the
|
|
186
|
+
real commands, and the narration is on the frame. It names the numbers of
|
|
187
|
+
this release. The older `pitch.mp4` stays where it was — a live recording
|
|
188
|
+
from a Mac, with a voice — and the README now says which is which.
|
|
189
|
+
|
|
190
|
+
### Fixed
|
|
191
|
+
|
|
192
|
+
- **A refresh overwrote a profile whose backup could not be written.** The
|
|
193
|
+
backup is the only copy of a file the user may have edited by hand — and
|
|
194
|
+
for an npm install that file is inside node_modules, where the next
|
|
195
|
+
`npm install` destroys it anyway. The error was swallowed and the
|
|
196
|
+
profile was overwritten regardless: the user's edits were gone, with no
|
|
197
|
+
backup and no message. A backup that cannot be written now stops the
|
|
198
|
+
refresh.
|
|
199
|
+
|
|
200
|
+
- **`refresh --all` reported profiles it never checked.** A profile that
|
|
201
|
+
could not be parsed was skipped without a word, and the summary counted
|
|
202
|
+
every id it had been given — so a broken profile was reported as a
|
|
203
|
+
refreshed one, and the withdrawn-model check never ran for it. The count
|
|
204
|
+
is of the profiles actually checked, the skipped ones are named, and the
|
|
205
|
+
command exits non-zero: a skipped profile is a failure, not a footnote.
|
|
206
|
+
|
|
207
|
+
- **`generateConfig(profile, { comments: false })` produced a file nothing
|
|
208
|
+
could read.** The whole header is comments, so turning comments off
|
|
209
|
+
emitted it without the `// ` prefix, in front of the JSON body. The
|
|
210
|
+
result was neither JSON nor JSONC, and the kit's own `parseJsonc`
|
|
211
|
+
returned null for it. It is pure JSON now.
|
|
212
|
+
|
|
213
|
+
- **A file named `toString.md` was reported as shadowing a lane.** The
|
|
214
|
+
check was `lane in (actual.agent ?? {})`, and `in` walks the prototype
|
|
215
|
+
chain — so a file named after a property every object has matched a lane
|
|
216
|
+
that does not exist, and `verify` reported a clean install as broken.
|
|
217
|
+
|
|
218
|
+
- **`--role` took the next flag as its value.** `oc-harness run --role
|
|
219
|
+
--max-rounds 5` set the role to the literal string `--max-rounds`, and
|
|
220
|
+
the loop then failed every round and marked every task blocked. A flag
|
|
221
|
+
with no value is refused now, and so is a role that is not a lane —
|
|
222
|
+
while the answer is still "that is not a lane".
|
|
223
|
+
|
|
224
|
+
- **The install deleted a file of yours that only looked like the kit's.**
|
|
225
|
+
The old agent directory was recognised by a substring, and a directory
|
|
226
|
+
called `space-harness/agents-backup` contains that substring. The file
|
|
227
|
+
was deleted. The match is on the parent directory now, segment by
|
|
228
|
+
segment.
|
|
229
|
+
|
|
230
|
+
- **A wrapper the list did not name was a way around the rules.** The
|
|
231
|
+
wrapper list held `env`, `sudo` and a handful of others, so the programs
|
|
232
|
+
that run a command as their argument - `timeout`, `nice`, `ionice`,
|
|
233
|
+
`stdbuf`, `setsid`, `chroot`, `taskset`, `busybox` - were commands the
|
|
234
|
+
rules had never seen. `timeout 5 rm -rf /tmp/x` deleted a tree, and
|
|
235
|
+
`timeout 5 npm publish` published a package, and neither was blocked.
|
|
236
|
+
|
|
237
|
+
The list is a table now, and the flags that take a value are the
|
|
238
|
+
WRAPPER'S OWN: `sudo -n` is a flag and the word after it is the command,
|
|
239
|
+
while `stdbuf -o 0` takes a mode and the word after it is the mode. One
|
|
240
|
+
list for all of them made `sudo -n rm -rf /tmp/x` read `rm` as the value
|
|
241
|
+
of `-n`, so the delete was invisible - which is the same hole in the other
|
|
242
|
+
direction.
|
|
243
|
+
|
|
244
|
+
- **`env -S "rm -rf /tmp/x"` was allowed.** `-S` takes a string and splits
|
|
245
|
+
it into words, so the string is a command line - the same thing `sh -c`
|
|
246
|
+
carries. It used to arrive as ONE token, and the command word was read
|
|
247
|
+
out of it as the text after the last slash, so the guard looked at `x`.
|
|
248
|
+
The payload is read now, and it is read the way `sh -c` is read.
|
|
249
|
+
|
|
250
|
+
- **`git -C/tmp reset --hard` was allowed.** `-C` and `-c` take a value, and
|
|
251
|
+
the value can be written against the flag. The resolver only knew the
|
|
252
|
+
spaced form, so the subcommand it found was the flag itself - and the
|
|
253
|
+
rule that watches for a history rewrite never saw the subcommand at all.
|
|
254
|
+
|
|
255
|
+
- **A public key was refused as a private key.** The rule named the key
|
|
256
|
+
FILES, and `id_rsa.pub` is the public half of `id_rsa` - a file the user
|
|
257
|
+
is meant to publish. Writing one was refused for a reason nobody could
|
|
258
|
+
find.
|
|
259
|
+
|
|
260
|
+
- **A file that was downloaded and then run was ordinary work.** The rule
|
|
261
|
+
caught `curl x | sh` and nothing else, so the same download in two steps
|
|
262
|
+
- fetch it to a file, then run the file - was allowed. It runs exactly the
|
|
263
|
+
same downloaded script.
|
|
264
|
+
|
|
265
|
+
- **A symlink out of the project was a write out of the project.** The
|
|
266
|
+
scope check compared paths as TEXT, so a link inside the project that
|
|
267
|
+
pointed at `/etc` was a path inside the project - and the write landed
|
|
268
|
+
outside it. The comparison is made on the real path now, and the part of
|
|
269
|
+
the path that does not exist yet is put back on the end.
|
|
270
|
+
|
|
271
|
+
- **`oc-harness agents` called a prototype property a lane.** The check was
|
|
272
|
+
`name in (config.agent)`, and `in` walks the prototype chain - so a project
|
|
273
|
+
agent file named `toString.md` or `constructor.md` matched a property every
|
|
274
|
+
object has: the file was reported as shadowing a lane that does not exist,
|
|
275
|
+
and the missing `model:` that was the point of the report was never
|
|
276
|
+
reported either. `verify` had the same check and the same fix; this one
|
|
277
|
+
was left behind.
|
|
278
|
+
|
|
279
|
+
- **`auto --json` dropped everything the refresh had found.** The refresh
|
|
280
|
+
prints what it found - a withdrawn model, a profile it could not read - and
|
|
281
|
+
`auto --json` silences the human output, so those findings went nowhere:
|
|
282
|
+
the report said `"ok": true` while a profile had never been checked
|
|
283
|
+
against the live catalogue, and the exit code said something else
|
|
284
|
+
entirely. The refresh hands its findings back now, and they are in the
|
|
285
|
+
report, in the step's status, and in the exit code.
|
|
286
|
+
|
|
287
|
+
- **`baseline --db=/path` measured the wrong database.** Only the spaced
|
|
288
|
+
form `--db /path` was read, so the `=` spelling - the one every other flag
|
|
289
|
+
in this kit accepts - was ignored, and the default database was measured
|
|
290
|
+
instead. The report then described a database nobody asked about. Both
|
|
291
|
+
spellings work now, and a `--db` with no value is refused instead of
|
|
292
|
+
crashing.
|
|
293
|
+
|
|
294
|
+
- **`doctor --check-models` reported a profile it never checked.** A profile
|
|
295
|
+
that could not be read was skipped without a word, and the run then said
|
|
296
|
+
every shipped model was still in the catalogue - so a profile that was NEVER
|
|
297
|
+
CHECKED was reported as one that was checked and found fine. It is the same
|
|
298
|
+
hole `refresh --all` had, in the one place that runs from a cron. The
|
|
299
|
+
skipped profiles are named now, and the run fails.
|
|
300
|
+
|
|
301
|
+
### Changed
|
|
302
|
+
|
|
303
|
+
- **The `verify` and `doc` lanes ask nothing.** Both ran the project's own
|
|
304
|
+
commands, and both asked before every one of them that the lane's
|
|
305
|
+
allowlist did not name. A run that stops to be asked is a run that does
|
|
306
|
+
not finish. The mechanical safety is unchanged: the guard plugin still
|
|
307
|
+
refuses the denied commands in code, and the lanes still cannot edit.
|
|
308
|
+
|
|
309
|
+
### Added
|
|
310
|
+
|
|
311
|
+
- **A recorder for a machine with no screen.** `vrec` drives the real
|
|
312
|
+
keyboard and the real screen, so it needs macOS, the Screen Recording
|
|
313
|
+
permission, and somebody to grant it. A headless box has none of those,
|
|
314
|
+
and it has no `ffmpeg` either, so the usual encode step has nothing to
|
|
315
|
+
encode with. The published demo was made with `vrec`; on such a machine
|
|
316
|
+
it cannot be made at all.
|
|
317
|
+
|
|
318
|
+
The recorder is in the tools repository, next to `vrec` and `btake` — one
|
|
319
|
+
directory up, and there is exactly one of each, because a copy inside
|
|
320
|
+
this repository is how two copies drift. It draws the scene on a canvas
|
|
321
|
+
and lets the browser be the encoder: Chromium's `captureStream` into
|
|
322
|
+
`MediaRecorder` at a fixed clock, which comes out as an `.mp4` with an
|
|
323
|
+
H.264 track and no ffmpeg on the machine at all. The text on screen is
|
|
324
|
+
the real output of the real commands — the take spec names the commands,
|
|
325
|
+
and they are run when the video is built.
|
|
326
|
+
|
|
327
|
+
The gate is the part that matters. A file that exists, has the right
|
|
328
|
+
duration and has the right size can still be thirty seconds of black, so
|
|
329
|
+
the gate decodes the take, pulls frames out of it, and looks at them: a
|
|
330
|
+
frame with no spread in its pixels shows nothing, and the take fails.
|
|
331
|
+
The frames are written out as PNGs, because a check you cannot look at
|
|
332
|
+
is a check you cannot argue with.
|
|
333
|
+
|
|
334
|
+
**The take is local and the video is local.** The take spec lives in
|
|
335
|
+
`video/takes/` and the recording lands in `video/build/`; both are
|
|
336
|
+
gitignored. The repository ships the recorder and not the take, and a
|
|
337
|
+
recording stays on the machine that made it until somebody decides,
|
|
338
|
+
deliberately, to publish it.
|
|
339
|
+
|
|
340
|
+
And the narration is on the screen. A voice is the best way to carry it
|
|
341
|
+
and it is not always there — `say` is macOS, and a headless Linux box
|
|
342
|
+
has no espeak, no festival and no cloud TTS — so the words go on the
|
|
343
|
+
frame, one caption at a time, and the take still says what it is about.
|
|
344
|
+
|
|
15
345
|
## [0.5.7] — 2026-10-21
|
|
16
346
|
|
|
17
347
|
Published. The tag is what cut it: `git tag v0.5.7 && git push origin main
|
package/README.md
CHANGED
|
@@ -101,7 +101,7 @@ The default profile is a **fleet**: ten jobs, ten free models, one session.
|
|
|
101
101
|
The table below is the **unrestricted** fleet (`fleet-all`), where every lane
|
|
102
102
|
gets the model built for it. The default is the **safe** fleet
|
|
103
103
|
(`fleet-safe`), where every lane resolves to a provider with a zero-retention
|
|
104
|
-
policy and
|
|
104
|
+
policy and seven of the ten run on a second-choice model to stay there. Both
|
|
105
105
|
cover the same ten jobs; they differ in what they are willing to promise
|
|
106
106
|
about your prompts.
|
|
107
107
|
|
|
@@ -116,7 +116,13 @@ about your prompts.
|
|
|
116
116
|
| `doc` | read a long document or a spec | GLM 5.3 Flash | 7.4 s |
|
|
117
117
|
| `schema` | produce JSON or a JSON schema | DeepSeek V4.1 Flash | 10.7 s |
|
|
118
118
|
| `sweep` | sweep a lot of code mechanically | Nemotron 3.5 Super VL | 8.4 s |
|
|
119
|
-
| `longform` | read a megabyte, or write alternatives |
|
|
119
|
+
| `longform` | read a megabyte, or write alternatives | MiniMax M3.1 Flash | 6.9 s |
|
|
120
|
+
|
|
121
|
+
`longform` used to run on `longcat-2.5-preview-free`, and the provider
|
|
122
|
+
withdrew it. A withdrawn model is not a lane, it is a hole in a lane: the
|
|
123
|
+
route falls to the next rung of the same chain, which is MiniMax - the only
|
|
124
|
+
model left on the tier with a million-token context. The `measured` column
|
|
125
|
+
for that lane is therefore MiniMax's, not the withdrawn model's.
|
|
120
126
|
|
|
121
127
|
The `measured` column is the median of five one-line requests to that model
|
|
122
128
|
on 2026-10-21 - the same prompt for every model, from the process start, so
|
|
@@ -241,7 +247,6 @@ it. One file covers ten jobs either way.
|
|
|
241
247
|
| `beam-free` | 5 | 1,048,576 | `max` | retained |
|
|
242
248
|
| `deepseek-v4.1-flash-free` | 5 | 200,000 | `max` | zero-retention |
|
|
243
249
|
| `glm-5.3-flash-free` | 5 | 200,000 | `max` | zero-retention |
|
|
244
|
-
| `longcat-2.5-preview-free` | 5 | 1,000,000 | — | zero-retention |
|
|
245
250
|
| `nemotron-3.5-super-vl-free` | 5 | 1,048,576 | `xhigh` | **may train** |
|
|
246
251
|
| `nemotron-3-ultra-free` | 5 | 1,000,000 | — | **may train** |
|
|
247
252
|
| `nemotron-3.5-lightning-free` | 5 | 262,144 | — | **may train** |
|
|
@@ -281,7 +286,7 @@ withdrawn, and one of them was the **default** at the time:
|
|
|
281
286
|
The full list, with a reason for each, is in
|
|
282
287
|
[`profiles/index.json`](profiles/index.json).
|
|
283
288
|
|
|
284
|
-
### The
|
|
289
|
+
### The five commands, and when each one runs
|
|
285
290
|
|
|
286
291
|
| command | what it checks | when |
|
|
287
292
|
|---|---|---|
|
|
@@ -289,6 +294,7 @@ The full list, with a reason for each, is in
|
|
|
289
294
|
| `oc-harness doctor` | every profile against the live model catalogue, and the installed config against the active profile | **before every release**, and whenever a lane starts failing with a 429 or a "model not found" |
|
|
290
295
|
| `oc-harness doctor --check-models` | one run, non-zero when a shipped model is no longer in the catalogue | from cron: the shape a scheduler wants |
|
|
291
296
|
| `oc-harness doctor --watch` | keeps watching, and reports only what **changed** | leave it in a terminal for a week |
|
|
297
|
+
| `oc-harness research` | what could **replace** a withdrawn model, and which lanes each candidate could serve | when `doctor` reports a withdrawal, and from cron before it happens |
|
|
292
298
|
| `oc-harness refresh <id>` | re-derives a profile's machine-derived fields from live metadata | when `doctor` reports drift |
|
|
293
299
|
| `oc-harness verify` | the installed config still matches the profile, and the kit's own files are still installed | after every `use` / `install` |
|
|
294
300
|
|
|
@@ -351,6 +357,193 @@ out to be. A lane is never left without one.
|
|
|
351
357
|
See [docs/KNOWN-LIMITS.md](docs/KNOWN-LIMITS.md) for the rest of what this
|
|
352
358
|
tier cannot do.
|
|
353
359
|
|
|
360
|
+
### Research: what can replace the model that just died
|
|
361
|
+
|
|
362
|
+
`doctor` answers one half of the question a perishable tier forces on you:
|
|
363
|
+
**is the model I depend on still there?** The other half is **what is there
|
|
364
|
+
instead** — and nobody was answering that half, so a withdrawal left a hole
|
|
365
|
+
only a person could fill, by hand, out of a catalogue dump.
|
|
366
|
+
|
|
367
|
+
`oc-harness research` is that half, and it is a file:
|
|
368
|
+
[`src/research.js`](src/research.js).
|
|
369
|
+
|
|
370
|
+
```
|
|
371
|
+
$ oc-harness research
|
|
372
|
+
|
|
373
|
+
research tier opencode, 10 live of 46 free
|
|
374
|
+
|
|
375
|
+
✓ every model in use is still live
|
|
376
|
+
|
|
377
|
+
lanes
|
|
378
|
+
build 2 live 0 dead
|
|
379
|
+
plan 2 live 0 dead
|
|
380
|
+
reviewer 2 live 0 dead
|
|
381
|
+
...
|
|
382
|
+
|
|
383
|
+
candidates live, free, and not in use yet
|
|
384
|
+
|
|
385
|
+
flint-40b-free ctx 262,144 out 65,536 effort low,medium,high,xhigh,max
|
|
386
|
+
fits: plan, build, reviewer, locate, verify, doc, schema, sweep, longform
|
|
387
|
+
retention: unknown
|
|
388
|
+
no retention verdict in retention.yml - a person has to read the provider's terms
|
|
389
|
+
|
|
390
|
+
nothing here is written anywhere. A retention verdict is a person reading a provider's terms.
|
|
391
|
+
```
|
|
392
|
+
|
|
393
|
+
That is the healthy state. When something has been withdrawn, the same command
|
|
394
|
+
says which, and says where it was used:
|
|
395
|
+
|
|
396
|
+
```
|
|
397
|
+
gone 1 model(s) in use have been withdrawn
|
|
398
|
+
|
|
399
|
+
✗ longcat-2.5-preview-free
|
|
400
|
+
withdrawn from the provider - the catalogue keeps the record and marks it deprecated
|
|
401
|
+
base model of longcat-2.5-preview-free
|
|
402
|
+
fallback rung in fleet-all: build, plan, locate, vision, doc, sweep, longform
|
|
403
|
+
fallback rung in fleet-safe: build, plan, reviewer, locate, verify, vision, doc, schema, sweep, longform
|
|
404
|
+
|
|
405
|
+
lanes
|
|
406
|
+
build 2 live 1 dead (longcat-2.5-preview-free)
|
|
407
|
+
plan 2 live 1 dead (longcat-2.5-preview-free)
|
|
408
|
+
...
|
|
409
|
+
```
|
|
410
|
+
|
|
411
|
+
It reports four things:
|
|
412
|
+
|
|
413
|
+
1. **The models in use that have been withdrawn**, split into the ones that are
|
|
414
|
+
a profile's own base model and the ones that are a rung in a lane chain. A
|
|
415
|
+
dead base model means that profile does nothing; a dead chain rung is a
|
|
416
|
+
fallback that fails on the day it is needed.
|
|
417
|
+
2. **The lanes of the default profile**, each with how many rungs of its chain
|
|
418
|
+
are still alive. A lane with zero live rungs is a lane with no model.
|
|
419
|
+
3. **The candidates** — live, free models the kit does not use yet, each with
|
|
420
|
+
its context window, its output cap, the effort levels it publishes, and
|
|
421
|
+
**which lanes it could serve**.
|
|
422
|
+
4. **What is still undecided** about each candidate.
|
|
423
|
+
|
|
424
|
+
#### Why the candidate list knows about lanes
|
|
425
|
+
|
|
426
|
+
A model is not "better" or "worse" in the abstract. It is better or worse
|
|
427
|
+
**for a lane**, and the lanes are not asking for the same thing:
|
|
428
|
+
|
|
429
|
+
| what the lane needs | why |
|
|
430
|
+
|---|---|
|
|
431
|
+
| tool calls | every lane calls tools, so every lane needs a model that can |
|
|
432
|
+
| an image in | the `vision` lane reads an image, so it needs a model that takes one |
|
|
433
|
+
| a published effort level | a lane whose role asks for `max` needs a model that publishes `max` |
|
|
434
|
+
|
|
435
|
+
So `research` matches each candidate against each lane, and a model that
|
|
436
|
+
publishes **no** effort levels counts as a match rather than a failure: there
|
|
437
|
+
is nothing to mismatch, and the kit already puts such models in the chains of
|
|
438
|
+
lanes that ask for one.
|
|
439
|
+
|
|
440
|
+
#### The line the script does not cross
|
|
441
|
+
|
|
442
|
+
`research` **writes nothing**. Not `retention.yml`, not a profile, not the
|
|
443
|
+
config.
|
|
444
|
+
|
|
445
|
+
That is the same line `refresh` draws, and it is drawn for the same reason:
|
|
446
|
+
`retention.yml` is a human judgement about a provider's terms — whether your
|
|
447
|
+
prompts are kept, and whether they train on them. A machine cannot read a
|
|
448
|
+
provider's terms of service and come back with a privacy verdict. So a
|
|
449
|
+
candidate with no verdict is reported as `retention: unknown` and carries an
|
|
450
|
+
explicit blocker, rather than being quietly filed under the tier it happens to
|
|
451
|
+
be cheapest on.
|
|
452
|
+
|
|
453
|
+
`refresh` draws the line in the other direction, and it is the one that
|
|
454
|
+
matters: it **only ever removes**. A model that has been withdrawn is removed
|
|
455
|
+
from the chains. A model that might do the job better is never added, because
|
|
456
|
+
putting a model into a lane changes what that lane does, and that is a
|
|
457
|
+
person's decision.
|
|
458
|
+
|
|
459
|
+
The output of `research` is a shortlist to argue with, not a decision to
|
|
460
|
+
accept. When you have read the terms and made the call, `retention.yml` is
|
|
461
|
+
where the call lives, and `doctor` checks the rest.
|
|
462
|
+
|
|
463
|
+
#### Run it on a schedule
|
|
464
|
+
|
|
465
|
+
```cron
|
|
466
|
+
0 9 * * * cd ~/.config/opencode && oc-harness doctor --check-models
|
|
467
|
+
30 9 * * * cd ~/.config/opencode && oc-harness research
|
|
468
|
+
```
|
|
469
|
+
|
|
470
|
+
The first line says a route died. The second says what to do about it, and it
|
|
471
|
+
runs half an hour later because by then you have had a cup of coffee and the
|
|
472
|
+
answer is a decision, not an alarm.
|
|
473
|
+
|
|
474
|
+
`research --json` prints one parseable object and nothing else, so a script
|
|
475
|
+
can read it:
|
|
476
|
+
|
|
477
|
+
```json
|
|
478
|
+
{
|
|
479
|
+
"ok": true,
|
|
480
|
+
"tier": { "provider": "opencode", "total": 46, "live": 10, "withdrawn": 36 },
|
|
481
|
+
"shipped": { "count": 9, "gone": [] },
|
|
482
|
+
"lanes": [ { "lane": "build", "dead": [], "live": ["minimax-m3.1-flash-free", "deepseek-v4.1-flash-free"] } ],
|
|
483
|
+
"candidates": [ { "id": "flint-40b-free", "fits": ["build"], "retention": "unknown", "blockers": ["..."] } ]
|
|
484
|
+
}
|
|
485
|
+
```
|
|
486
|
+
|
|
487
|
+
#### One thing the catalogue does that a naive check misses
|
|
488
|
+
|
|
489
|
+
**A withdrawn model stays in the catalogue.** models.dev does not delete a
|
|
490
|
+
model the provider pulls — it keeps the record and marks it
|
|
491
|
+
`status: "deprecated"`. So "is it still in the catalogue" and "can it still
|
|
492
|
+
run" are two different questions, and a check that asks the first one reports
|
|
493
|
+
a dead model as a healthy fallback.
|
|
494
|
+
|
|
495
|
+
That is not hypothetical. This kit's own default fleet carried a dead fallback
|
|
496
|
+
rung in **all ten lanes**, and `doctor` called it healthy, because the chain
|
|
497
|
+
check asked whether the model was still present. It now asks whether it is
|
|
498
|
+
still alive, and it says so:
|
|
499
|
+
|
|
500
|
+
```
|
|
501
|
+
! fleet-safe drift
|
|
502
|
+
lane build: longcat-2.5-preview-free withdrawn, falls back to minimax-m3.1-flash-free
|
|
503
|
+
```
|
|
504
|
+
|
|
505
|
+
**Free is a price, not a name.** The tier is the set of models the provider
|
|
506
|
+
charges nothing for. `big-pickle` is on it and does not end in `-free`, so
|
|
507
|
+
the suffix is a naming convention and not the test — `research` reads the
|
|
508
|
+
price.
|
|
509
|
+
|
|
510
|
+
### Refresh: taking the dead rung out
|
|
511
|
+
|
|
512
|
+
`research` tells you a rung is dead. `refresh` takes it out.
|
|
513
|
+
|
|
514
|
+
```
|
|
515
|
+
$ oc-harness refresh --all
|
|
516
|
+
|
|
517
|
+
refreshed 11 of 11 profile(s) against the live catalogue
|
|
518
|
+
~ fleet-safe: withdrawn rungs removed: longcat-2.5-preview-free
|
|
519
|
+
lane build: minimax, deepseek, longcat -> minimax, deepseek
|
|
520
|
+
lane longform: longcat, minimax, deepseek -> minimax, deepseek
|
|
521
|
+
...
|
|
522
|
+
```
|
|
523
|
+
|
|
524
|
+
A chain names the models that could serve a lane, best first. A model the
|
|
525
|
+
provider has withdrawn is not one of them any more: it cannot run, so it is
|
|
526
|
+
dead weight in a chain whose entire job is to survive a withdrawal. The
|
|
527
|
+
refresh removes it, and if the lane's own primary was the dead one, the lane
|
|
528
|
+
takes the next live rung as its primary — a chain has to start with its
|
|
529
|
+
model. The fallback queue loses its dead entries, and the utility route moves
|
|
530
|
+
to a model the profile still reaches.
|
|
531
|
+
|
|
532
|
+
**The refresh only ever REMOVES.** Putting a model into a lane changes what
|
|
533
|
+
that lane does, and that is a person's decision — it is what `research` is
|
|
534
|
+
for. A script that quietly added a candidate would be putting a model whose
|
|
535
|
+
terms nobody has read into the path of every prompt.
|
|
536
|
+
|
|
537
|
+
Two cases a script cannot repair, and says so instead of pretending:
|
|
538
|
+
|
|
539
|
+
- **A profile whose own model was withdrawn.** There is no chain behind it to
|
|
540
|
+
fall back to, so there is nothing to repair. `refresh` reports it and leaves
|
|
541
|
+
the file alone; the honest fix is to retire the profile, and retiring one is
|
|
542
|
+
a decision a person makes. (That is exactly what happened to
|
|
543
|
+
`longcat-2.5-preview-free`.)
|
|
544
|
+
- **A lane left with no model at all.** Every rung of its chain was dead. It
|
|
545
|
+
is reported, not filled.
|
|
546
|
+
|
|
354
547
|
## Requirements
|
|
355
548
|
|
|
356
549
|
| | |
|
|
@@ -358,9 +551,9 @@ tier cannot do.
|
|
|
358
551
|
| Node | **22 or newer** — `engines` in package.json, and the CLI uses `node:` prefixed builtins. Node 20 fails at import time, not with a friendly message. |
|
|
359
552
|
| npm | 10 or newer, for `npm ci` when installing from a checkout |
|
|
360
553
|
| OS | macOS, Linux and Windows. The tests run on `ubuntu-latest`. |
|
|
361
|
-
| Shell | none needed to *use* the kit — no `bash`, `zsh` or `fish` setup. `doctor` and `
|
|
362
|
-
| Network | only `doctor` and `
|
|
363
|
-
| Disk | `use` and `install` write into `~/.config/opencode/` and nothing else. `doctor` writes nothing. `refresh` rewrites a profile in the kit's own
|
|
554
|
+
| Shell | none needed to *use* the kit — no `bash`, `zsh` or `fish` setup. `doctor`, `refresh` and `research` call `curl` to read models.dev. `baseline` and the measurement scripts read the install database with `sqlite3`, and fall back to python3's own `sqlite3` module when that binary is not installed — so python3 is the one thing to have. Everything else, including `install`, `use` and `verify`, works offline. |
|
|
555
|
+
| Network | only `doctor`, `refresh` and `research` need it, and only to read the live model catalogue. `list` is offline: it reads the shipped profiles. Everything else works offline. |
|
|
556
|
+
| Disk | `use` and `install` write into `~/.config/opencode/` and nothing else. `doctor` writes nothing. `research` writes nothing — it is a report, not a migration. `refresh` rewrites a profile in the kit's own
|
|
364
557
|
directory - `node_modules` for an npm install - and keeps a backup of the
|
|
365
558
|
one it replaced. |
|
|
366
559
|
|
|
@@ -375,6 +568,8 @@ keeps a dated copy in `~/.config/opencode/backups/`, including under
|
|
|
375
568
|
oc-harness list profiles with retention and caveats
|
|
376
569
|
oc-harness show <id> resolved lanes, derived numbers, fallback chains, guard rules
|
|
377
570
|
oc-harness init pick a free profile, then install + use + verify
|
|
571
|
+
oc-harness auto refresh + install + use + verify, asking nothing
|
|
572
|
+
oc-harness auto --json the same reconcile, as a report a script can read
|
|
378
573
|
oc-harness use <id> compile a profile into ~/.config/opencode/opencode.jsonc
|
|
379
574
|
oc-harness use <id> --dryrun print a summary of what would be written, write nothing
|
|
380
575
|
oc-harness refresh <id> re-derive a profile against the live model list
|
|
@@ -571,10 +766,23 @@ recent calls was a window on *recency*, and a long session walks out of it:
|
|
|
571
766
|
after six more calls, the only copy of a file the agent had read was gone,
|
|
572
767
|
and the agent read it again. On the long-task A/B that was seventeen
|
|
573
768
|
re-reads and five times the input tokens. The newest copy of each distinct
|
|
574
|
-
output now survives
|
|
575
|
-
|
|
576
|
-
|
|
577
|
-
|
|
769
|
+
output now survives, and only the repeats are stubbed - **-69% input tokens
|
|
770
|
+
and -60% wall time** on that task, at the cost of a bigger context. It is a
|
|
771
|
+
setting: `prune.keepDistinctOutputs` in the profile, `false` to get the old
|
|
772
|
+
behaviour back.
|
|
773
|
+
|
|
774
|
+
**The distinct outputs have a ceiling.** The rule without one is not a
|
|
775
|
+
trade-off but a leak: it keeps every output the session has ever produced,
|
|
776
|
+
and a long session grows without limit. Measured on a real 137-message
|
|
777
|
+
session on 2026-10-22: the request was 5.26 MB, the pruner removed 0.04 MB of
|
|
778
|
+
it (0.8%), and 4.74 MB of it - 92% - was the distinct outputs of the whole
|
|
779
|
+
session. The profile's own budget for that request was 176,000 characters,
|
|
780
|
+
so the request was twenty-nine times over. The ceiling keeps the newest
|
|
781
|
+
distinct outputs and stubs the ones past it, so the property the rule was
|
|
782
|
+
added for - the copy the agent is working with survives, so it does not read
|
|
783
|
+
it again - still holds. It is `prune.keepDistinctBudgetChars` in the profile;
|
|
784
|
+
unset means the profile's own context budget, and `0` keeps no distinct
|
|
785
|
+
output at all.
|
|
578
786
|
|
|
579
787
|
## Memory
|
|
580
788
|
|
|
@@ -677,6 +885,28 @@ oc-harness install # writes everything that file points at
|
|
|
677
885
|
`GLOBAL.md` into `~/.config/opencode/`, so this repo stays the single source
|
|
678
886
|
of truth and an edit here takes effect on the next opencode start.
|
|
679
887
|
|
|
888
|
+
### The one command that asks nothing
|
|
889
|
+
|
|
890
|
+
`init` is the front door for a person: it asks which free profile to run on.
|
|
891
|
+
`auto` is the one for a job, an agent, or a script — it reconciles the whole
|
|
892
|
+
install and never waits for an answer:
|
|
893
|
+
|
|
894
|
+
```bash
|
|
895
|
+
oc-harness auto # refresh + install + use + verify
|
|
896
|
+
oc-harness auto --json # the same, as a report a script can read
|
|
897
|
+
oc-harness auto --offline # skip the refresh, for a machine that cannot reach models.dev
|
|
898
|
+
```
|
|
899
|
+
|
|
900
|
+
The profile is the one the installed config names, or the default. A file of
|
|
901
|
+
yours that the installer would overwrite is **left alone and reported**, not
|
|
902
|
+
overwritten and not asked about — the answer a person would give.
|
|
903
|
+
|
|
904
|
+
`--yes` still means consent. A profile whose models may train on your prompts
|
|
905
|
+
is refused without it, exactly as `use` refuses it: the automation is about
|
|
906
|
+
the mechanics, not about the consent. `auto` exits non-zero when `verify`
|
|
907
|
+
finds a problem it is not allowed to repair, and `--json` puts the problems in
|
|
908
|
+
the report.
|
|
909
|
+
|
|
680
910
|
Then restart opencode. Check it took:
|
|
681
911
|
|
|
682
912
|
```bash
|