opencode-harness-kit 0.4.2 → 0.4.4
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +544 -19
- package/CONTRIBUTING.md +242 -0
- package/GLOBAL.md +34 -5
- package/MIGRATION.md +151 -0
- package/README.md +294 -37
- package/SECURITY.md +30 -0
- package/bin/baseline.mjs +15 -7
- package/bin/git-hooks/pre-commit +41 -9
- package/bin/oc-harness.mjs +1265 -82
- package/dist/config.d.ts +127 -0
- package/dist/denylist.d.ts +33 -0
- package/dist/effort.d.ts +64 -0
- package/dist/guard.d.ts +23 -0
- package/dist/health.d.ts +43 -0
- package/dist/index.d.ts +9 -0
- package/dist/index.js +532 -149
- package/dist/plugin.d.ts +15 -0
- package/dist/profile.d.ts +399 -0
- package/dist/project.d.ts +8 -0
- package/dist/prune.d.ts +6 -0
- package/dist/router.d.ts +3 -0
- package/dist/rules.d.ts +109 -0
- package/dist/safe.d.ts +7 -0
- package/dist/telemetry-store.d.ts +12 -0
- package/dist/telemetry.d.ts +3 -0
- package/docs/EXIT-CODES.md +71 -0
- package/docs/KNOWN-LIMITS.md +45 -21
- package/docs/RESEARCH.md +9 -4
- package/docs/ROUTING.md +53 -8
- package/eval/RESULTS.md +443 -0
- package/eval/ab.mjs +743 -0
- package/eval/attribute.mjs +170 -0
- package/eval/flags.mjs +88 -0
- package/eval/measure-install.mjs +323 -0
- package/eval/measure-prune.mjs +339 -0
- package/eval/metrics.js +79 -0
- package/eval/premises.mjs +200 -0
- package/eval/prepare.mjs +233 -0
- package/eval/run.mjs +376 -0
- package/eval/tasks-self.json +81 -0
- package/eval/tasks-ts.json +50 -0
- package/eval/tasks.json +83 -0
- package/eval/thresholds.mjs +134 -0
- package/eval/verify.mjs +287 -0
- package/package.json +44 -17
- package/profiles/fleet-all.json +6 -1
- package/profiles/fleet-safe.json +7 -3
- package/profiles/schema.json +182 -51
- package/prompts/build.md +4 -2
- package/skills/space-build/SKILL.md +4 -2
- package/skills/space-finish/SKILL.md +6 -2
- package/skills/space-plan/reference/task-template.md +13 -5
- package/src/config.js +182 -32
- package/src/denylist.js +95 -0
- package/src/driver/run.mjs +81 -16
- package/src/effort.js +15 -4
- package/src/guard.ts +165 -13
- package/src/health.js +80 -0
- package/src/index.ts +15 -4
- package/src/lane-usage.js +72 -18
- package/src/plugin-type-check.ts +38 -0
- package/src/plugin.ts +39 -0
- package/src/profile.js +225 -26
- package/src/project.js +47 -0
- package/src/prune.ts +110 -31
- package/src/router.ts +32 -63
- package/src/rules.js +99 -7
- package/src/safe.ts +27 -0
- package/src/sqlite.js +144 -0
- package/src/tasks.js +112 -29
- package/src/telemetry-store.ts +62 -0
- package/src/telemetry.ts +60 -57
- package/src/trust.js +70 -9
package/CHANGELOG.md
CHANGED
|
@@ -12,7 +12,538 @@ is not happening.
|
|
|
12
12
|
|
|
13
13
|
---
|
|
14
14
|
|
|
15
|
-
## [
|
|
15
|
+
## [0.4.4] — 2026-10-21
|
|
16
|
+
|
|
17
|
+
Published. The tag is what cut it: `git tag v0.4.4 && git push origin main
|
|
18
|
+
v0.4.4`, and the publish job watches for exactly that.
|
|
19
|
+
|
|
20
|
+
This release is mostly about things that were quietly wrong. The guard let a
|
|
21
|
+
wrapper through, the grader scored correct answers wrong, the A/B compared
|
|
22
|
+
two different models, and the published type declarations did not resolve for
|
|
23
|
+
a consumer. None of them announced itself: they were quiet, and a quiet
|
|
24
|
+
failure in a harness is the kind that gets believed.
|
|
25
|
+
|
|
26
|
+
### Fixed
|
|
27
|
+
|
|
28
|
+
**The denylist was matched against the raw command string, so a wrapper got
|
|
29
|
+
past it.**
|
|
30
|
+
|
|
31
|
+
`cd /tmp && npm publish`, `sudo npm publish` and `sh -c "npm publish"` all
|
|
32
|
+
went through. The patterns are `npm publish*`, and the command does not
|
|
33
|
+
start with them. The kit already had the machinery for exactly this - the
|
|
34
|
+
rules are matched per segment, with the wrappers stripped and the payload of
|
|
35
|
+
a `bash -c` followed - and the denylist now uses it. One list, read one way.
|
|
36
|
+
|
|
37
|
+
**The grader paired a path with a line number that belonged to something
|
|
38
|
+
else.**
|
|
39
|
+
|
|
40
|
+
A gap that could cross a sentence turned "the answer is in src/a.js.
|
|
41
|
+
Unrelated note: see line 2 of the report" into a claim about line 2. The
|
|
42
|
+
bare path was then dropped, because the file had been cited with a line, and
|
|
43
|
+
a right file was graded against a line the agent never claimed. The gap may
|
|
44
|
+
no longer contain a full stop, and a number on the next line has to be
|
|
45
|
+
introduced by a label - which is the shape `**File:**` / `**Line:**` takes.
|
|
46
|
+
|
|
47
|
+
**A citation to the build output was graded as evidence about the source.**
|
|
48
|
+
|
|
49
|
+
`dist/`, `eval/` and `test/` are in the exclusion list, and the test was
|
|
50
|
+
`endsWith` - which never matches a directory, because a file path does not
|
|
51
|
+
end with a directory name. The exclusion now matches a directory prefix and
|
|
52
|
+
a file name, wherever the file is.
|
|
53
|
+
|
|
54
|
+
**The A/B compared two different models and called the difference the kit's
|
|
55
|
+
cost.**
|
|
56
|
+
|
|
57
|
+
The kit-off arm ran with an empty config, so opencode picked its own default
|
|
58
|
+
model; the kit-on arm ran on the model in the installed kit's config. The
|
|
59
|
+
default it picked did not answer either - four kit-off runs hit the
|
|
60
|
+
600-second ceiling with no output at all, and a timed-out run has zero tools
|
|
61
|
+
and zero context, so it entered the medians as a fast, empty, wrong answer.
|
|
62
|
+
|
|
63
|
+
The model is now pinned in the kit-off home, read from the config the same
|
|
64
|
+
way the kit reads it: the lane model of the default agent, which is the
|
|
65
|
+
model opencode actually runs and the one the router moves when a route is
|
|
66
|
+
spent. The report prints it, so a reader can check that both arms ran on
|
|
67
|
+
the same one.
|
|
68
|
+
|
|
69
|
+
**The published type declarations did not resolve for a consumer.**
|
|
70
|
+
|
|
71
|
+
They import a type from `@opencode-ai/plugin`, which was a devDependency, so
|
|
72
|
+
a consumer that installed the package type-checked against a type it did
|
|
73
|
+
not have. The subpath exports - `opencode-harness-kit/profile` and the rest
|
|
74
|
+
- declared no types at all, even though the declarations were built and
|
|
75
|
+
shipped.
|
|
76
|
+
|
|
77
|
+
The plugin package depends on `effect`, `zod` and `@opencode-ai/sdk`, and
|
|
78
|
+
has three peer dependencies of its own, so making it a peer is not an
|
|
79
|
+
answer either: an optional peer is never installed by npm, and a required
|
|
80
|
+
one drags all of that onto a consumer that only wanted to compile a config.
|
|
81
|
+
|
|
82
|
+
So the published declarations no longer import it. The plugin type is
|
|
83
|
+
declared structurally, in `src/plugin.ts`, and the real type is checked
|
|
84
|
+
against it in `src/plugin-type-check.ts` - in CI, where the package is
|
|
85
|
+
installed. A drift between the two is a failed build, not a broken
|
|
86
|
+
consumer.
|
|
87
|
+
|
|
88
|
+
**`sudo bash -c "rm -rf /"` was not blocked.**
|
|
89
|
+
|
|
90
|
+
The guard unwrapped a shell's payload only when the shell was the first
|
|
91
|
+
word of the segment, so a wrapper in front of it - `sudo`, `env`, `xargs`,
|
|
92
|
+
`npx` - hid the whole command line. The same shape let `npx npm publish`
|
|
93
|
+
through. The wrappers come off first now.
|
|
94
|
+
|
|
95
|
+
**A failed catalogue fetch ended the watch instead of continuing it.**
|
|
96
|
+
|
|
97
|
+
`fetchCatalog` called `fail()`, which exits. The catch around it could
|
|
98
|
+
never run, so `doctor --watch` died on the first failed poll - a transient
|
|
99
|
+
network error turned a monitor into a dead one. The fetch throws now, and
|
|
100
|
+
the callers decide: the watch carries on, and `doctor --json` prints a
|
|
101
|
+
JSON object with the error in it rather than nothing at all.
|
|
102
|
+
|
|
103
|
+
**`uninstall` left the kit's own broken links behind.**
|
|
104
|
+
|
|
105
|
+
It checked whether a link pointed inside the kit by resolving it, and
|
|
106
|
+
resolving a DANGLING link throws - which is exactly the case the command
|
|
107
|
+
exists for, because `npm uninstall` leaves four dangling links behind. A
|
|
108
|
+
link is now resolved without requiring its target to exist, and the same
|
|
109
|
+
test is applied to `GLOBAL.md` and the plugin link, which were removed
|
|
110
|
+
whatever they pointed at.
|
|
111
|
+
|
|
112
|
+
**`oc-harness cost` reported 0 for a project that had pruned anything.**
|
|
113
|
+
|
|
114
|
+
The field in the log was renamed from `savedBytes` to `savedChars`, and
|
|
115
|
+
the log is project-local and persistent: every project that upgraded had
|
|
116
|
+
the old records and read them as zero. The old name is still read.
|
|
117
|
+
|
|
118
|
+
**The published type declarations did not resolve for a consumer.**
|
|
119
|
+
|
|
120
|
+
They import a type from `@opencode-ai/plugin`, which was a devDependency, so
|
|
121
|
+
a consumer that installed the package type-checked against a type it did
|
|
122
|
+
not have. The subpath exports - `opencode-harness-kit/profile` and the rest
|
|
123
|
+
- declared no types at all, even though the declarations were built and
|
|
124
|
+
shipped. The plugin is now an optional peer dependency and every subpath
|
|
125
|
+
declares its types.
|
|
126
|
+
|
|
127
|
+
**`oc-harness cost` reported characters as bytes.**
|
|
128
|
+
|
|
129
|
+
The pruner counts characters, and the number it printed said "bytes", which
|
|
130
|
+
is wrong by a factor on any text that is not ASCII.
|
|
131
|
+
|
|
132
|
+
**`refresh --all` wrote its backups into `profiles/`.**
|
|
133
|
+
|
|
134
|
+
That directory is published to npm, so every refresh added a file to the
|
|
135
|
+
tarball, and nothing ever pruned them. The backups go to the config
|
|
136
|
+
directory now, with the other backups.
|
|
137
|
+
|
|
138
|
+
**`uninstall` deleted the user's own symlinks.**
|
|
139
|
+
|
|
140
|
+
It removed every symlink in `prompts/` and `skills/`, on the grounds that a
|
|
141
|
+
directory of symlinks is the kit's. A user with their own symlink in that
|
|
142
|
+
directory lost it. A link is the kit's when it points inside the kit.
|
|
143
|
+
|
|
144
|
+
**`doctor --json` put colour codes inside the messages**, and the exit code
|
|
145
|
+
for a missing config or a missing profile changed without the change being
|
|
146
|
+
said out loud. The codes are out now, and an error in JSON mode is a JSON
|
|
147
|
+
object.
|
|
148
|
+
|
|
149
|
+
**In a project that is not a git repository, the kit did nothing at all.**
|
|
150
|
+
|
|
151
|
+
`ctx.worktree` is opencode's worktree: the git root of the project. For a
|
|
152
|
+
directory that is not a git repository it is the FILESYSTEM ROOT, and the kit
|
|
153
|
+
wrote its files under it. So in every non-git project the config hook's write
|
|
154
|
+
to `.space/active` failed with a permission error that the plugin swallowed,
|
|
155
|
+
and then:
|
|
156
|
+
|
|
157
|
+
- the pruner found no active profile and did nothing at all,
|
|
158
|
+
- the router never recorded a spent route, so no lane ever failed over,
|
|
159
|
+
- the telemetry went nowhere,
|
|
160
|
+
- and the guard's scope check was against "/", which contains every path, so
|
|
161
|
+
it was not a scope check.
|
|
162
|
+
|
|
163
|
+
None of it was visible from the code, because the kit had only ever been run
|
|
164
|
+
in git repositories. It was found by running a session in a directory that is
|
|
165
|
+
not one and noticing that no `.space/` appeared. The A/B had been running
|
|
166
|
+
against the broken kit: its subject is a `git archive`, so it has no `.git`,
|
|
167
|
+
and the pruner was inert for every kit-on run. The measurement this
|
|
168
|
+
repository exists to make was measuring a kit that did not prune.
|
|
169
|
+
|
|
170
|
+
The project root is now resolved from the session's own DIRECTORY, walking up
|
|
171
|
+
to the git root when there is one. A subdirectory of a monorepo is still that
|
|
172
|
+
repository, and a directory that is not a repository is its own project.
|
|
173
|
+
|
|
174
|
+
**The grader scored correct answers wrong.** An agent that answers in a
|
|
175
|
+
markdown field writes the file on one line and the number on the next:
|
|
176
|
+
|
|
177
|
+
**File:** `src/profile.js`
|
|
178
|
+
**Line:** 105
|
|
179
|
+
|
|
180
|
+
and the citation pattern stopped at the newline, so the pair was never made
|
|
181
|
+
and the answer was scored wrong. It is the shape most of the answers took:
|
|
182
|
+
on one measured run, four correct answers graded as one. The gap may cross a
|
|
183
|
+
newline now, and the file/line field form is paired explicitly rather than by
|
|
184
|
+
proximity. The numbers in `eval/RESULTS.md` were computed with this grader,
|
|
185
|
+
so they are being re-measured rather than patched.
|
|
186
|
+
|
|
187
|
+
**`npm publish` was ALLOWED.** The allowlist had a blanket `npm*`, so
|
|
188
|
+
publishing to a registry needed no prompt and no question, while GLOBAL.md
|
|
189
|
+
says to ask before publishing. The mechanical layer contradicted the
|
|
190
|
+
judgement layer, which is the exact failure this kit is built to catch.
|
|
191
|
+
|
|
192
|
+
The list now lives in `src/denylist.js` because it is enforced in TWO places:
|
|
193
|
+
by opencode's permission map, and by the guard plugin. The two used to be
|
|
194
|
+
different lists - the config denied `gh pr create` and `npm publish` and the
|
|
195
|
+
guard blocked neither - so the config's own comment, "the guard plugin
|
|
196
|
+
enforces the same set in code", was false. A permission map can be overridden
|
|
197
|
+
by a plugin or lost to a hand-edit; the guard does not care.
|
|
198
|
+
|
|
199
|
+
`npm install` and `npm ci` are no longer allowed without a question. They
|
|
200
|
+
run code the agent did not read, and they change the tree the agent was
|
|
201
|
+
asked to change.
|
|
202
|
+
|
|
203
|
+
**The subject of the A/B held the grader and the answer key.** The subject of
|
|
204
|
+
the self suite is this repository, so it held `eval/verify.mjs` - the grader -
|
|
205
|
+
and `eval/tasks-self.json`, the answer key. An agent that found the grader
|
|
206
|
+
used it to check its own work, and one that found the suite read the answer.
|
|
207
|
+
Both were measured: an agent that read the grader made 48 tool calls to a
|
|
208
|
+
task that needed 5. `eval/` and `test/` are excluded from the subject now.
|
|
209
|
+
The answer lives in `src/`, and so does the question.
|
|
210
|
+
|
|
211
|
+
**The kit's own cost, measured and cut.** The A/B said the kit was a net loss
|
|
212
|
+
on a short task, and the commands showed why: the agent was running the
|
|
213
|
+
project's whole test suite, and installing the dependencies, to rename one
|
|
214
|
+
constant. The rule that told it to do that was absolute.
|
|
215
|
+
|
|
216
|
+
It is scoped now: run the check that covers what you changed, run the one the
|
|
217
|
+
task named, and do not re-run a check that already passed. Do not install
|
|
218
|
+
anything the task did not ask for. And "locate before you read" is
|
|
219
|
+
conditional - when the question already names the file, read it.
|
|
220
|
+
|
|
221
|
+
Measured over 20 runs after the change, against 60 before:
|
|
222
|
+
|
|
223
|
+
| per run | before | after |
|
|
224
|
+
|---|---|---|
|
|
225
|
+
| `npm test` calls | 2.2 | **0.9** |
|
|
226
|
+
| `npm install` calls | 0.4 | **0** |
|
|
227
|
+
|
|
228
|
+
### Fixed, again
|
|
229
|
+
|
|
230
|
+
A review of this release found six more, and fixing them turned up two of the
|
|
231
|
+
same class.
|
|
232
|
+
|
|
233
|
+
**The project-root fix was incomplete.** Two call sites still read the
|
|
234
|
+
worktree: the effort hook looked for the spent-route file under it, and the
|
|
235
|
+
router wrote the spent-route file under it. So the router recorded a spent
|
|
236
|
+
route where the effort hook was not looking, and a lane the router had
|
|
237
|
+
already given up on was put back onto it.
|
|
238
|
+
|
|
239
|
+
**The documented profile override had never worked.**
|
|
240
|
+
`OPENCODE_HARNESS_PROFILE` is a profile ID, and it was being checked with
|
|
241
|
+
`existsSync`, like the marker file beside it. It never matched a file, so
|
|
242
|
+
the override that the docs and the measurement scripts both rely on was
|
|
243
|
+
inert - and the measurement it gates reported 0% removed and failed.
|
|
244
|
+
|
|
245
|
+
**The measurement depended on the developer's local state.** The pruner
|
|
246
|
+
finds its profile from `.space/active` in the project, and that file is
|
|
247
|
+
gitignored, so on a fresh clone there is none, the pruner does nothing, and
|
|
248
|
+
the gate says the kit removes nothing. The environment override is set for
|
|
249
|
+
the measurement now, and the measurement runs against a scratch directory
|
|
250
|
+
rather than writing into the repository it is measuring.
|
|
251
|
+
|
|
252
|
+
**A gate that could be switched off by a typo.** `--min-saved abc` became
|
|
253
|
+
NaN, and `NaN` compares false against everything, so the gate never failed.
|
|
254
|
+
The script uses the shared flag parser, which throws.
|
|
255
|
+
|
|
256
|
+
**The A/B stopped the whole experiment twice, both times wrongly.** Once on a
|
|
257
|
+
scratch file in the temp directory, which is the one place GLOBAL.md tells an
|
|
258
|
+
agent to put scratch files and the one place the guard's scope rule allows;
|
|
259
|
+
and once on an agent that grepped the filesystem root. A READ outside the
|
|
260
|
+
subject is wasteful and sometimes a privacy leak, but it damages nothing - the
|
|
261
|
+
pilot that mattered WROTE to the real repository. A write outside the target
|
|
262
|
+
stops the experiment; a read outside it is counted and reported.
|
|
263
|
+
|
|
264
|
+
**A citation to the suite's own file was accepted as evidence.** The self
|
|
265
|
+
suite's subject is a copy of this repository, so the answer key is inside
|
|
266
|
+
it, and an agent that greps for the symbol could cite the suite and pass
|
|
267
|
+
without reading the source. The suite files are excluded from the citations.
|
|
268
|
+
|
|
269
|
+
**The failover threw away the utility route.** `applySessionModel` set
|
|
270
|
+
`small_model` to the fallback profile's session model, so the fleet's cheap
|
|
271
|
+
route was replaced exactly when the fleet failed over - and `verify` then
|
|
272
|
+
called the difference between the file on disk and the running session a
|
|
273
|
+
drift.
|
|
274
|
+
|
|
275
|
+
Also: the python fallback ran every query twice, the CLI check spawned a
|
|
276
|
+
process per query, the database path was not URI-escaped, `--since` was
|
|
277
|
+
ignored as the last argument, `--interval abc` became a 1ms loop, a failed
|
|
278
|
+
catalogue fetch in watch mode announced every missing model as "is back", a
|
|
279
|
+
non-list `plugin` value was replaced instead of kept, the `why` in
|
|
280
|
+
fleet-all's utility route claimed the model was in every chain when it is in
|
|
281
|
+
seven of ten, and the install-wide measurement could report a 100% context
|
|
282
|
+
improvement from a NULL average.
|
|
283
|
+
|
|
284
|
+
### Changed
|
|
285
|
+
|
|
286
|
+
- **The install-wide measurement is an observation, and there is a controlled
|
|
287
|
+
one.** `measure:check` compared the work before the kit landed with the work
|
|
288
|
+
after, and gated on the context per request. On this install it said the
|
|
289
|
+
context per request went UP by 16%, with twice as many reads and twice as
|
|
290
|
+
many messages in the second window - a number about the jobs, not about the
|
|
291
|
+
kit. The after-window was also "everything since the install", so the
|
|
292
|
+
comparison was a different one every time the script ran.
|
|
293
|
+
|
|
294
|
+
The windows are fixed and equal around the split now, and the report says
|
|
295
|
+
how long they were. The context drop is still reported and still gated, but
|
|
296
|
+
only when the two windows are comparable: when one side did twice the work
|
|
297
|
+
of the other, the number is skipped with a note that says why, because a gate
|
|
298
|
+
that fails for reasons nobody can act on is a gate people switch off.
|
|
299
|
+
|
|
300
|
+
The gate uses the number that IS comparable. opencode stores the message
|
|
301
|
+
parts UNPRUNED - the pruner rewrites the outgoing request, not the history
|
|
302
|
+
- so the stored parts of a real session are exactly what the pruner was
|
|
303
|
+
given, and replaying the real pruner over them measures exactly what it
|
|
304
|
+
removed, on the same messages with and without. On this install: **80.5% of
|
|
305
|
+
the context across 60 real sessions**, 6,608 parts stubbed - the number
|
|
306
|
+
`npm run measure:prune` prints.
|
|
307
|
+
|
|
308
|
+
The measurement scripts also run without the sqlite3 CLI now. python3's
|
|
309
|
+
built-in sqlite3 module does the same job, and python3 is on more machines
|
|
310
|
+
than the CLI is.
|
|
311
|
+
|
|
312
|
+
- **`small_model` is a cheaper route.** It is what opencode uses for its short
|
|
313
|
+
auxiliary calls - a title, a one-line summary - and it was the session model,
|
|
314
|
+
so every one of those calls paid the session model's REASONING. Measured on
|
|
315
|
+
this tier, a title costs 49 reasoning tokens on deepseek and 332 on
|
|
316
|
+
nemotron-3.5-super-vl, and deepseek answers in 7.2s against 22.7s for glm. A
|
|
317
|
+
fleet names a cheaper route now and the generated file says which one; a
|
|
318
|
+
single-model profile keeps its one route, because "one route, and it is the
|
|
319
|
+
one you asked for" is a promise those profiles make.
|
|
320
|
+
|
|
321
|
+
- **The safe fleet honours every tier request it makes.** The tier is worth
|
|
322
|
+
17x on reasoning tokens - deepseek spends 320 a message at `max` and 19 at
|
|
323
|
+
`low` - so a lane that asks for a tier and lands on a model that publishes
|
|
324
|
+
none runs at that model's natural rate instead. One lane was genuinely
|
|
325
|
+
expensive: `sweep` is a bulk mechanical search, so reasoning on it is pure
|
|
326
|
+
waste, and it ran on a model that reasons 420 tokens a message at 16.6s.
|
|
327
|
+
No zero-retention model publishes a `none` level, so the lane could not even
|
|
328
|
+
turn reasoning off when it asked to. It runs on deepseek now: 19 reasoning
|
|
329
|
+
tokens at 3.4s.
|
|
330
|
+
|
|
331
|
+
The three lanes in the unrestricted fleet that cannot get their tier turn
|
|
332
|
+
out to be the three CHEAPEST reasoners on the tier - 116, 59 and 18 tokens a
|
|
333
|
+
message, against 289 for `build` and 198 for `plan`, which keep their tier
|
|
334
|
+
on purpose. The tier is the lever, and it is not where the tokens are. The
|
|
335
|
+
generated config says so where it happens, in the form
|
|
336
|
+
`locate=unset(asked low)`.
|
|
337
|
+
|
|
338
|
+
- `npm run check` is the offline gate — build, typecheck, tests — and is green
|
|
339
|
+
on a checkout that has never seen an OpenCode login. The live half moved to
|
|
340
|
+
`npm run check:install`, which needs a machine that has run OpenCode. The
|
|
341
|
+
old claim that `npm run check` had to pass on an untouched checkout was
|
|
342
|
+
false, and it was the reason the whole gate was skipped.
|
|
343
|
+
- `profiles/schema.json` no longer accepts `roleModels.<lane>.levels`. The
|
|
344
|
+
field was in the schema, validated as an array, and then read by nothing, so
|
|
345
|
+
a profile author could fill it with a lie and get no error.
|
|
346
|
+
- `oc-harness show` now prints what the guard blocks and why. It was the one
|
|
347
|
+
part of the kit a user could not inspect at all.
|
|
348
|
+
- The config directory path uses `homedir()` in all three places that build
|
|
349
|
+
it. One of them used `process.env.HOME`, which with HOME unset produced a
|
|
350
|
+
RELATIVE path, so `use` wrote to one place and the reader read another.
|
|
351
|
+
- `retention.yml` is cached like the lane table next to it. It was re-read and
|
|
352
|
+
re-parsed for every candidate of every lane.
|
|
353
|
+
- The severity table ranked `unknown` as milder than `may-train`, while
|
|
354
|
+
`retentionAllowed` refuses `unknown` outright. The two contradicted each
|
|
355
|
+
other, and the table is the one that decides what a profile is labelled.
|
|
356
|
+
- The `lanes` header printed the raw record count and called it "lane
|
|
357
|
+
request(s)". A request is one assistant message, and a message is written
|
|
358
|
+
twice, so the headline said twice what the table under it said.
|
|
359
|
+
- The router checked the health file for a profile ID where the file is keyed
|
|
360
|
+
by MODEL. For the ten single-model profiles the two are the same string and
|
|
361
|
+
the check worked by accident; for the two fleets it could never fire.
|
|
362
|
+
- `harness.fallback` in the generated config was a copy of the first id of a
|
|
363
|
+
queue the router reads straight from the profile file, in a key nothing
|
|
364
|
+
reads. `fallbackProfileId` was a parameter no caller ever supplied.
|
|
365
|
+
- The README quoted two different sets of numbers for the same measurement.
|
|
366
|
+
There is one set now, and it is the one in `docs/RESEARCH.md` and
|
|
367
|
+
`eval/RESULTS.md`.
|
|
368
|
+
- Both `.space/telemetry.jsonl` and `.space/prune.jsonl` are written in the
|
|
369
|
+
project. The prune log used to land in the process's working directory,
|
|
370
|
+
which is often your home directory.
|
|
371
|
+
### Added
|
|
372
|
+
|
|
373
|
+
- **`show --json` names the profile's own notes `notes`** and the compiler's
|
|
374
|
+
adjustments `adjustments`, because the human report calls them different
|
|
375
|
+
things and the JSON should not make one name mean two.
|
|
376
|
+
|
|
377
|
+
- **`doctor --json` and `show --json`.** "Is the install healthy" and "what
|
|
378
|
+
does this profile resolve to" are questions a CI job and a dashboard ask,
|
|
379
|
+
and the answers were tables of padded columns with colour codes in them.
|
|
380
|
+
Both now print the machine-readable form and nothing else - not the human
|
|
381
|
+
report first and the JSON after it, which is not machine-readable. An
|
|
382
|
+
error in JSON mode is a JSON object, so a script that pipes the output
|
|
383
|
+
into a parser gets something it can read either way. `lanes --json` used
|
|
384
|
+
the key `lanes` for the USAGE of each lane, and `show --json` uses it for
|
|
385
|
+
the lane TABLE; the usage one is now `usage`, so one name does not mean
|
|
386
|
+
two shapes.
|
|
387
|
+
|
|
388
|
+
- **`docs/EXIT-CODES.md`**, because a command that can fail should say how,
|
|
389
|
+
and "non-zero" does not tell a script whether the work landed or there was
|
|
390
|
+
no task file. Four codes for the loop, two for the doctor, and the rule
|
|
391
|
+
that a measurement which could not be compared is reported and NOT gated.
|
|
392
|
+
|
|
393
|
+
- **`MIGRATION.md` covers 0.3 to 0.4.** The upgrade asks something of the
|
|
394
|
+
user - the guard now refuses in code what it used to only suggest, and the
|
|
395
|
+
pre-commit hook runs the project's own tests instead of the kit's - so it
|
|
396
|
+
is written down, with the command to check it and the command to undo it.
|
|
397
|
+
|
|
398
|
+
- **`oc-harness cost`** — what the kit is doing on THIS project, in numbers.
|
|
399
|
+
The kit's claim is a trade: a fixed overhead on every request, and the
|
|
400
|
+
context it prunes back. Both numbers were already being written — the prune
|
|
401
|
+
log records what each pass removed, and the telemetry records what each lane
|
|
402
|
+
cost — and nothing read them together. On the one measured here: 83.7
|
|
403
|
+
million characters removed over 272 prune passes, 48,122 tool outputs
|
|
404
|
+
stubbed, and the lanes sorted by what they cost.
|
|
405
|
+
|
|
406
|
+
- **`oc-harness uninstall`.** The install SYMLINKS the plugin, the prompts,
|
|
407
|
+
the skills and GLOBAL.md out of the kit's directory, so `npm uninstall` left
|
|
408
|
+
four dangling symlinks and opencode then loaded neither the rules nor the
|
|
409
|
+
guard — and said nothing about it.
|
|
410
|
+
|
|
411
|
+
- **`oc-harness refresh --all`**, which re-derives every shipped profile
|
|
412
|
+
against the live catalogue, and updates `verified`, which is the field that
|
|
413
|
+
says when a profile was last checked. Leaving it stale made it a lie.
|
|
414
|
+
|
|
415
|
+
- **`oc-harness upgrade`**, which checks the registry for a newer kit,
|
|
416
|
+
refreshes the profiles, and writes the config again. Three things go stale
|
|
417
|
+
on their own and all three were invisible.
|
|
418
|
+
|
|
419
|
+
- **`oc-harness backups` and `oc-harness restore`.** The single `.bak` was
|
|
420
|
+
overwritten on the next `use` and carried no profile id, so "go back to
|
|
421
|
+
what I had" was not something a person could do at 2am. The config also
|
|
422
|
+
records which kit wrote it.
|
|
423
|
+
|
|
424
|
+
- **Type declarations.** The package's `types` pointed at `src/index.ts` — a
|
|
425
|
+
TypeScript SOURCE file — and the tsconfig had `noEmit`, so no declarations
|
|
426
|
+
were ever produced. A consumer type-checked the kit's source out of
|
|
427
|
+
node_modules.
|
|
428
|
+
|
|
429
|
+
- **Tests for the pieces the suite never reached**: the python fallback, the
|
|
430
|
+
project root for a project that is not a git repository, the escape rule,
|
|
431
|
+
the flag parser, and the measurement scripts themselves — the gate that
|
|
432
|
+
`npm run measure:check` depends on had no test at all.
|
|
433
|
+
|
|
434
|
+
- **The demo is back on the front page.** `docs/demo/pitch.mp4` and
|
|
435
|
+
`docs/demo-loop.gif` are in the README again, and `video/` still holds the
|
|
436
|
+
take script, the voiceover and the recording driver. The film was recorded
|
|
437
|
+
against an earlier version of the kit, and the README now says so next to
|
|
438
|
+
it rather than leaving a front page that quietly misstates the project's own
|
|
439
|
+
numbers - which is the reason 0.4.3 removed it in the first place, and the
|
|
440
|
+
reason it should not have been removed without putting the film back.
|
|
441
|
+
|
|
442
|
+
- **`OPENCODE_HARNESS_CATALOG_URL` overrides where the live model catalogue
|
|
443
|
+
comes from.** A mirror is one use; a local fixture is another, and a test
|
|
444
|
+
that reached models.dev would fail on a machine with no network. The `file://`
|
|
445
|
+
spelling is read directly, so a fixture needs no external binary.
|
|
446
|
+
|
|
447
|
+
- **`OC_HARNESS_NO_HOOK` skips the pre-commit hook.** A test that writes an
|
|
448
|
+
executable git hook into the repository it is testing is a test that changes
|
|
449
|
+
the repository.
|
|
450
|
+
|
|
451
|
+
- **`oc-harness use --memory` turns on a memory plugin.** The kit does not
|
|
452
|
+
implement memory, and it should not: a local vector store with embeddings, a
|
|
453
|
+
web UI, auto-capture and a user profile is a large thing to build badly, and
|
|
454
|
+
`opencode-mem` already is one. The entry is ADDED to the plugin list, so the
|
|
455
|
+
user's own plugins survive and a config that already has it does not grow a
|
|
456
|
+
second copy.
|
|
457
|
+
|
|
458
|
+
- **`oc-harness doctor --check-models` and `--watch`.** The question a
|
|
459
|
+
free-tier user actually has to ask is "is one of my routes gone", and it is
|
|
460
|
+
not the same question as "has a profile drifted". `--check-models` answers it
|
|
461
|
+
in one run with a non-zero exit, which is the shape a cron wants; `--watch`
|
|
462
|
+
keeps running and reports only what CHANGED.
|
|
463
|
+
|
|
464
|
+
- **`eval/measure-prune.mjs`.** The attributable measurement of the pruner,
|
|
465
|
+
described above.
|
|
466
|
+
|
|
467
|
+
- **The A/B suite is reproducible, and it has been run.** The old one ran
|
|
468
|
+
against `lablab-harness@8a674f5`, which is not public. A result nobody can
|
|
469
|
+
check is not a result. The new suite runs against the kit's own tree at a
|
|
470
|
+
pinned commit, so `git clone && git checkout <sha>` reproduces it exactly.
|
|
471
|
+
|
|
472
|
+
Ten reps per arm, 120 agent runs, and the result is not flattering:
|
|
473
|
+
|
|
474
|
+
| metric (median) | kit on | kit off |
|
|
475
|
+
|---|---|---|
|
|
476
|
+
| **correct** | **52/60** | **59/59** |
|
|
477
|
+
| tool calls | 11.5 | 3 |
|
|
478
|
+
| input tokens | 51,449 | 4,551 |
|
|
479
|
+
| final context | 39,603 | 10,855 |
|
|
480
|
+
| wall seconds | 43.6 | 32.5 |
|
|
481
|
+
| paths written outside the target | 0 | 0 |
|
|
482
|
+
|
|
483
|
+
**On a task that fits in one read, the kit is a cost.** Its instructions
|
|
484
|
+
cost eleven times the input tokens and four times the tool calls, the run
|
|
485
|
+
takes a third longer, and the answers are worse. The rule that says *locate
|
|
486
|
+
before you read* is a waste on a one-file question, and the tool counts show
|
|
487
|
+
it. The two edit tasks are a tie at 10/10, and the guard stopped no writes
|
|
488
|
+
in either arm.
|
|
489
|
+
|
|
490
|
+
The value the kit does have is in long sessions - the pruner removes 80.5%
|
|
491
|
+
of the context across real ones - and in process, which a grader that
|
|
492
|
+
counts correct answers cannot see. Whether it is worth it depends on
|
|
493
|
+
whether the work is long. **A repo of one-file lookups should not install
|
|
494
|
+
this.**
|
|
495
|
+
|
|
496
|
+
The full table, and what it does not establish, is in
|
|
497
|
+
`eval/RESULTS.md`.
|
|
498
|
+
|
|
499
|
+
- **`oc-harness refresh --all`** re-derives every shipped profile against the
|
|
500
|
+
live catalogue, and reports which models are gone.
|
|
501
|
+
- **`oc-harness doctor --check-models` and `--watch`** record a withdrawal in
|
|
502
|
+
`~/.config/opencode/withdrawn.jsonl`, with the profiles that used it. A model
|
|
503
|
+
is withdrawn without a rename and without a notice, so the only way to know
|
|
504
|
+
WHEN a lane started failing is to have been looking.
|
|
505
|
+
- **`oc-harness upgrade`** checks the registry for a newer kit, refreshes the
|
|
506
|
+
profiles, and writes the config again. Three things go stale on their own -
|
|
507
|
+
the kit, the profiles and the config - and all three were invisible.
|
|
508
|
+
- **A scheduled workflow** (`.github/workflows/measure.yml`), so the numbers
|
|
509
|
+
are kept rather than measured once. The offline half runs on every push;
|
|
510
|
+
the A/B runs nightly, because it costs model quota and takes about an hour.
|
|
511
|
+
- **`OPENCODE_HARNESS_PROFILES`** points the kit at a different profile
|
|
512
|
+
directory. A test that refreshes a profile would otherwise rewrite the kit's
|
|
513
|
+
own data - which is what happened the first time this was tried.
|
|
514
|
+
- **A flag appended to an npm script now overrides the script's own**:
|
|
515
|
+
`npm run measure:prune -- --sessions 1` means one session, not sixty.
|
|
516
|
+
|
|
517
|
+
- **Tests for the pieces the suite never reached**: the python fallback, the
|
|
518
|
+
project root for a project that is not a git repository, the escape rule,
|
|
519
|
+
and the flag parser.
|
|
520
|
+
|
|
521
|
+
### Security
|
|
522
|
+
|
|
523
|
+
- `eval/prepare.mjs` could delete a directory tree. `--name` was joined onto
|
|
524
|
+
the scratch root without being checked and the result removed with
|
|
525
|
+
`rmSync(..., { recursive: true })`, so `--name ../victim` deleted the
|
|
526
|
+
victim's tree. The label is a name and not a path, and the resolved path is
|
|
527
|
+
checked a second time against the scratch root.
|
|
528
|
+
- `eval/prepare.mjs` could run a command. The archive step was one `sh -c`
|
|
529
|
+
string and the paths were quoted with `JSON.stringify`, which is not shell
|
|
530
|
+
quoting — a path containing `$(...)` was executed by the shell. `git` and
|
|
531
|
+
`tar` are called with argument arrays now, and the tar goes through a pipe.
|
|
532
|
+
|
|
533
|
+
## [0.4.3] — 2026-10-20
|
|
534
|
+
|
|
535
|
+
### Removed
|
|
536
|
+
|
|
537
|
+
- **The video pipeline and the published demo.** The `video/` directory held
|
|
538
|
+
the take script, the voiceover, the recording driver and the superseded
|
|
539
|
+
dubbing pipeline; `docs/demo/pitch.mp4` and `docs/demo-loop.gif` held the
|
|
540
|
+
published film. This repository is a harness kit, and it is code.
|
|
541
|
+
|
|
542
|
+
The film was also stale: it said "seven profiles" and "one hundred and
|
|
543
|
+
sixty five tests", and the kit now ships twelve profiles and two hundred
|
|
544
|
+
and ten tests. A front page that misstates the project's own numbers is
|
|
545
|
+
worse than no front page, so it goes rather than gets quietly outdated
|
|
546
|
+
again.
|
|
16
547
|
|
|
17
548
|
## [0.4.2] — 2026-10-20
|
|
18
549
|
|
|
@@ -42,20 +573,11 @@ is not happening.
|
|
|
42
573
|
that can be opened in the JSON body. The list asked for JSON and worked; the
|
|
43
574
|
create did not, and failed with a message that said nothing about why. Every
|
|
44
575
|
npm trust call asks for JSON now.
|
|
45
|
-
-
|
|
46
|
-
|
|
47
|
-
|
|
48
|
-
|
|
49
|
-
|
|
50
|
-
browser. The failure looked like "npm wants something else now" and nothing
|
|
51
|
-
like the cause.
|
|
52
|
-
- **The URLs come from the response body, not from a guess.** The done URL
|
|
53
|
-
was rebuilt from the id, which sent a real request to the registry with a
|
|
54
|
-
made-up id; the registry answered "forbidden", which is not a token, so the
|
|
55
|
-
wait ran its full length and then reported that nobody had answered.
|
|
56
|
-
- **`OC_TRUST_NO_BROWSER` turns the browser off.** A test that opens a
|
|
57
|
-
browser on the machine it runs on is a test that breaks whatever the
|
|
58
|
-
person at the keyboard was doing.
|
|
576
|
+
- The per-command two-factor challenge, the done-URL guess and
|
|
577
|
+
`OC_TRUST_NO_BROWSER` were all fixed in 0.3.9 and are listed there. They
|
|
578
|
+
appear here once more because this release was cut from a branch that
|
|
579
|
+
predated 0.3.9's changelog entry, and a reader of the two releases was
|
|
580
|
+
left with the impression that the same three fixes happened twice.
|
|
59
581
|
|
|
60
582
|
### The release path, end to end
|
|
61
583
|
|
|
@@ -167,8 +689,9 @@ The publish is now automatic and holds no secret:
|
|
|
167
689
|
short-lived publish token, scoped to the workflow that asked for it. There
|
|
168
690
|
is no long-lived token in the repository to leak, rotate or expire - and npm
|
|
169
691
|
is revoking the legacy automation tokens anyway.
|
|
170
|
-
- The job
|
|
171
|
-
|
|
692
|
+
- The job runs on node 24, whose bundled npm already speaks trusted
|
|
693
|
+
publishing. An `npm install -g npm@latest` self-upgrade also works, and is
|
|
694
|
+
documented elsewhere as intermittently corrupting the install.
|
|
172
695
|
- The registry is configured by hand instead of through `setup-node`'s
|
|
173
696
|
`registry-url`, because that helper always writes an auth line, and an
|
|
174
697
|
empty auth line makes npm believe it is authenticated and skip the OIDC
|
|
@@ -381,7 +904,7 @@ the tool-output caps, and the `scout` → `locate` agent rename.
|
|
|
381
904
|
`git log --grep push` and `echo push` are not. A heredoc body is data, not
|
|
382
905
|
commands.
|
|
383
906
|
- **`tool_output` was pinned at 200 lines / 8 KB**, about 6× tighter than
|
|
384
|
-
OpenCode's default. It truncated 415 of 2,966 `read` calls and
|
|
907
|
+
OpenCode's default. It truncated 415 of 2,966 `read` calls and 423 calls
|
|
385
908
|
were issued with an `offset` to fetch the rest: the truncation *causes* the
|
|
386
909
|
re-read, one for one, at a measured 17.3 s median per request. The cap is
|
|
387
910
|
now the default, and pruning rather than truncation bounds the context.
|
|
@@ -433,7 +956,9 @@ silent in the worst way:
|
|
|
433
956
|
|
|
434
957
|
---
|
|
435
958
|
|
|
436
|
-
[Unreleased]: https://github.com/pham-canh/space-harness/compare/v0.4.
|
|
959
|
+
[Unreleased]: https://github.com/pham-canh/space-harness/compare/v0.4.3...HEAD
|
|
960
|
+
[0.4.4]: https://github.com/pham-canh/space-harness/compare/v0.4.3...v0.4.4
|
|
961
|
+
[0.4.3]: https://github.com/pham-canh/space-harness/compare/v0.4.2...v0.4.3
|
|
437
962
|
[0.4.2]: https://github.com/pham-canh/space-harness/compare/v0.4.1...v0.4.2
|
|
438
963
|
[0.4.1]: https://github.com/pham-canh/space-harness/compare/v0.4.0...v0.4.1
|
|
439
964
|
[0.4.0]: https://github.com/pham-canh/space-harness/compare/v0.3.9...v0.4.0
|