opencode-harness-kit 0.4.2 → 0.4.4

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (73) hide show
  1. package/CHANGELOG.md +544 -19
  2. package/CONTRIBUTING.md +242 -0
  3. package/GLOBAL.md +34 -5
  4. package/MIGRATION.md +151 -0
  5. package/README.md +294 -37
  6. package/SECURITY.md +30 -0
  7. package/bin/baseline.mjs +15 -7
  8. package/bin/git-hooks/pre-commit +41 -9
  9. package/bin/oc-harness.mjs +1265 -82
  10. package/dist/config.d.ts +127 -0
  11. package/dist/denylist.d.ts +33 -0
  12. package/dist/effort.d.ts +64 -0
  13. package/dist/guard.d.ts +23 -0
  14. package/dist/health.d.ts +43 -0
  15. package/dist/index.d.ts +9 -0
  16. package/dist/index.js +532 -149
  17. package/dist/plugin.d.ts +15 -0
  18. package/dist/profile.d.ts +399 -0
  19. package/dist/project.d.ts +8 -0
  20. package/dist/prune.d.ts +6 -0
  21. package/dist/router.d.ts +3 -0
  22. package/dist/rules.d.ts +109 -0
  23. package/dist/safe.d.ts +7 -0
  24. package/dist/telemetry-store.d.ts +12 -0
  25. package/dist/telemetry.d.ts +3 -0
  26. package/docs/EXIT-CODES.md +71 -0
  27. package/docs/KNOWN-LIMITS.md +45 -21
  28. package/docs/RESEARCH.md +9 -4
  29. package/docs/ROUTING.md +53 -8
  30. package/eval/RESULTS.md +443 -0
  31. package/eval/ab.mjs +743 -0
  32. package/eval/attribute.mjs +170 -0
  33. package/eval/flags.mjs +88 -0
  34. package/eval/measure-install.mjs +323 -0
  35. package/eval/measure-prune.mjs +339 -0
  36. package/eval/metrics.js +79 -0
  37. package/eval/premises.mjs +200 -0
  38. package/eval/prepare.mjs +233 -0
  39. package/eval/run.mjs +376 -0
  40. package/eval/tasks-self.json +81 -0
  41. package/eval/tasks-ts.json +50 -0
  42. package/eval/tasks.json +83 -0
  43. package/eval/thresholds.mjs +134 -0
  44. package/eval/verify.mjs +287 -0
  45. package/package.json +44 -17
  46. package/profiles/fleet-all.json +6 -1
  47. package/profiles/fleet-safe.json +7 -3
  48. package/profiles/schema.json +182 -51
  49. package/prompts/build.md +4 -2
  50. package/skills/space-build/SKILL.md +4 -2
  51. package/skills/space-finish/SKILL.md +6 -2
  52. package/skills/space-plan/reference/task-template.md +13 -5
  53. package/src/config.js +182 -32
  54. package/src/denylist.js +95 -0
  55. package/src/driver/run.mjs +81 -16
  56. package/src/effort.js +15 -4
  57. package/src/guard.ts +165 -13
  58. package/src/health.js +80 -0
  59. package/src/index.ts +15 -4
  60. package/src/lane-usage.js +72 -18
  61. package/src/plugin-type-check.ts +38 -0
  62. package/src/plugin.ts +39 -0
  63. package/src/profile.js +225 -26
  64. package/src/project.js +47 -0
  65. package/src/prune.ts +110 -31
  66. package/src/router.ts +32 -63
  67. package/src/rules.js +99 -7
  68. package/src/safe.ts +27 -0
  69. package/src/sqlite.js +144 -0
  70. package/src/tasks.js +112 -29
  71. package/src/telemetry-store.ts +62 -0
  72. package/src/telemetry.ts +60 -57
  73. package/src/trust.js +70 -9
package/CHANGELOG.md CHANGED
@@ -12,7 +12,538 @@ is not happening.
12
12
 
13
13
  ---
14
14
 
15
- ## [Unreleased]
15
+ ## [0.4.4] — 2026-10-21
16
+
17
+ Published. The tag is what cut it: `git tag v0.4.4 && git push origin main
18
+ v0.4.4`, and the publish job watches for exactly that.
19
+
20
+ This release is mostly about things that were quietly wrong. The guard let a
21
+ wrapper through, the grader scored correct answers wrong, the A/B compared
22
+ two different models, and the published type declarations did not resolve for
23
+ a consumer. None of them announced itself: they were quiet, and a quiet
24
+ failure in a harness is the kind that gets believed.
25
+
26
+ ### Fixed
27
+
28
+ **The denylist was matched against the raw command string, so a wrapper got
29
+ past it.**
30
+
31
+ `cd /tmp && npm publish`, `sudo npm publish` and `sh -c "npm publish"` all
32
+ went through. The patterns are `npm publish*`, and the command does not
33
+ start with them. The kit already had the machinery for exactly this - the
34
+ rules are matched per segment, with the wrappers stripped and the payload of
35
+ a `bash -c` followed - and the denylist now uses it. One list, read one way.
36
+
37
+ **The grader paired a path with a line number that belonged to something
38
+ else.**
39
+
40
+ A gap that could cross a sentence turned "the answer is in src/a.js.
41
+ Unrelated note: see line 2 of the report" into a claim about line 2. The
42
+ bare path was then dropped, because the file had been cited with a line, and
43
+ a right file was graded against a line the agent never claimed. The gap may
44
+ no longer contain a full stop, and a number on the next line has to be
45
+ introduced by a label - which is the shape `**File:**` / `**Line:**` takes.
46
+
47
+ **A citation to the build output was graded as evidence about the source.**
48
+
49
+ `dist/`, `eval/` and `test/` are in the exclusion list, and the test was
50
+ `endsWith` - which never matches a directory, because a file path does not
51
+ end with a directory name. The exclusion now matches a directory prefix and
52
+ a file name, wherever the file is.
53
+
54
+ **The A/B compared two different models and called the difference the kit's
55
+ cost.**
56
+
57
+ The kit-off arm ran with an empty config, so opencode picked its own default
58
+ model; the kit-on arm ran on the model in the installed kit's config. The
59
+ default it picked did not answer either - four kit-off runs hit the
60
+ 600-second ceiling with no output at all, and a timed-out run has zero tools
61
+ and zero context, so it entered the medians as a fast, empty, wrong answer.
62
+
63
+ The model is now pinned in the kit-off home, read from the config the same
64
+ way the kit reads it: the lane model of the default agent, which is the
65
+ model opencode actually runs and the one the router moves when a route is
66
+ spent. The report prints it, so a reader can check that both arms ran on
67
+ the same one.
68
+
69
+ **The published type declarations did not resolve for a consumer.**
70
+
71
+ They import a type from `@opencode-ai/plugin`, which was a devDependency, so
72
+ a consumer that installed the package type-checked against a type it did
73
+ not have. The subpath exports - `opencode-harness-kit/profile` and the rest
74
+ - declared no types at all, even though the declarations were built and
75
+ shipped.
76
+
77
+ The plugin package depends on `effect`, `zod` and `@opencode-ai/sdk`, and
78
+ has three peer dependencies of its own, so making it a peer is not an
79
+ answer either: an optional peer is never installed by npm, and a required
80
+ one drags all of that onto a consumer that only wanted to compile a config.
81
+
82
+ So the published declarations no longer import it. The plugin type is
83
+ declared structurally, in `src/plugin.ts`, and the real type is checked
84
+ against it in `src/plugin-type-check.ts` - in CI, where the package is
85
+ installed. A drift between the two is a failed build, not a broken
86
+ consumer.
87
+
88
+ **`sudo bash -c "rm -rf /"` was not blocked.**
89
+
90
+ The guard unwrapped a shell's payload only when the shell was the first
91
+ word of the segment, so a wrapper in front of it - `sudo`, `env`, `xargs`,
92
+ `npx` - hid the whole command line. The same shape let `npx npm publish`
93
+ through. The wrappers come off first now.
94
+
95
+ **A failed catalogue fetch ended the watch instead of continuing it.**
96
+
97
+ `fetchCatalog` called `fail()`, which exits. The catch around it could
98
+ never run, so `doctor --watch` died on the first failed poll - a transient
99
+ network error turned a monitor into a dead one. The fetch throws now, and
100
+ the callers decide: the watch carries on, and `doctor --json` prints a
101
+ JSON object with the error in it rather than nothing at all.
102
+
103
+ **`uninstall` left the kit's own broken links behind.**
104
+
105
+ It checked whether a link pointed inside the kit by resolving it, and
106
+ resolving a DANGLING link throws - which is exactly the case the command
107
+ exists for, because `npm uninstall` leaves four dangling links behind. A
108
+ link is now resolved without requiring its target to exist, and the same
109
+ test is applied to `GLOBAL.md` and the plugin link, which were removed
110
+ whatever they pointed at.
111
+
112
+ **`oc-harness cost` reported 0 for a project that had pruned anything.**
113
+
114
+ The field in the log was renamed from `savedBytes` to `savedChars`, and
115
+ the log is project-local and persistent: every project that upgraded had
116
+ the old records and read them as zero. The old name is still read.
117
+
118
+ **The published type declarations did not resolve for a consumer.**
119
+
120
+ They import a type from `@opencode-ai/plugin`, which was a devDependency, so
121
+ a consumer that installed the package type-checked against a type it did
122
+ not have. The subpath exports - `opencode-harness-kit/profile` and the rest
123
+ - declared no types at all, even though the declarations were built and
124
+ shipped. The plugin is now an optional peer dependency and every subpath
125
+ declares its types.
126
+
127
+ **`oc-harness cost` reported characters as bytes.**
128
+
129
+ The pruner counts characters, and the number it printed said "bytes", which
130
+ is wrong by a factor on any text that is not ASCII.
131
+
132
+ **`refresh --all` wrote its backups into `profiles/`.**
133
+
134
+ That directory is published to npm, so every refresh added a file to the
135
+ tarball, and nothing ever pruned them. The backups go to the config
136
+ directory now, with the other backups.
137
+
138
+ **`uninstall` deleted the user's own symlinks.**
139
+
140
+ It removed every symlink in `prompts/` and `skills/`, on the grounds that a
141
+ directory of symlinks is the kit's. A user with their own symlink in that
142
+ directory lost it. A link is the kit's when it points inside the kit.
143
+
144
+ **`doctor --json` put colour codes inside the messages**, and the exit code
145
+ for a missing config or a missing profile changed without the change being
146
+ said out loud. The codes are out now, and an error in JSON mode is a JSON
147
+ object.
148
+
149
+ **In a project that is not a git repository, the kit did nothing at all.**
150
+
151
+ `ctx.worktree` is opencode's worktree: the git root of the project. For a
152
+ directory that is not a git repository it is the FILESYSTEM ROOT, and the kit
153
+ wrote its files under it. So in every non-git project the config hook's write
154
+ to `.space/active` failed with a permission error that the plugin swallowed,
155
+ and then:
156
+
157
+ - the pruner found no active profile and did nothing at all,
158
+ - the router never recorded a spent route, so no lane ever failed over,
159
+ - the telemetry went nowhere,
160
+ - and the guard's scope check was against "/", which contains every path, so
161
+ it was not a scope check.
162
+
163
+ None of it was visible from the code, because the kit had only ever been run
164
+ in git repositories. It was found by running a session in a directory that is
165
+ not one and noticing that no `.space/` appeared. The A/B had been running
166
+ against the broken kit: its subject is a `git archive`, so it has no `.git`,
167
+ and the pruner was inert for every kit-on run. The measurement this
168
+ repository exists to make was measuring a kit that did not prune.
169
+
170
+ The project root is now resolved from the session's own DIRECTORY, walking up
171
+ to the git root when there is one. A subdirectory of a monorepo is still that
172
+ repository, and a directory that is not a repository is its own project.
173
+
174
+ **The grader scored correct answers wrong.** An agent that answers in a
175
+ markdown field writes the file on one line and the number on the next:
176
+
177
+ **File:** `src/profile.js`
178
+ **Line:** 105
179
+
180
+ and the citation pattern stopped at the newline, so the pair was never made
181
+ and the answer was scored wrong. It is the shape most of the answers took:
182
+ on one measured run, four correct answers graded as one. The gap may cross a
183
+ newline now, and the file/line field form is paired explicitly rather than by
184
+ proximity. The numbers in `eval/RESULTS.md` were computed with this grader,
185
+ so they are being re-measured rather than patched.
186
+
187
+ **`npm publish` was ALLOWED.** The allowlist had a blanket `npm*`, so
188
+ publishing to a registry needed no prompt and no question, while GLOBAL.md
189
+ says to ask before publishing. The mechanical layer contradicted the
190
+ judgement layer, which is the exact failure this kit is built to catch.
191
+
192
+ The list now lives in `src/denylist.js` because it is enforced in TWO places:
193
+ by opencode's permission map, and by the guard plugin. The two used to be
194
+ different lists - the config denied `gh pr create` and `npm publish` and the
195
+ guard blocked neither - so the config's own comment, "the guard plugin
196
+ enforces the same set in code", was false. A permission map can be overridden
197
+ by a plugin or lost to a hand-edit; the guard does not care.
198
+
199
+ `npm install` and `npm ci` are no longer allowed without a question. They
200
+ run code the agent did not read, and they change the tree the agent was
201
+ asked to change.
202
+
203
+ **The subject of the A/B held the grader and the answer key.** The subject of
204
+ the self suite is this repository, so it held `eval/verify.mjs` - the grader -
205
+ and `eval/tasks-self.json`, the answer key. An agent that found the grader
206
+ used it to check its own work, and one that found the suite read the answer.
207
+ Both were measured: an agent that read the grader made 48 tool calls to a
208
+ task that needed 5. `eval/` and `test/` are excluded from the subject now.
209
+ The answer lives in `src/`, and so does the question.
210
+
211
+ **The kit's own cost, measured and cut.** The A/B said the kit was a net loss
212
+ on a short task, and the commands showed why: the agent was running the
213
+ project's whole test suite, and installing the dependencies, to rename one
214
+ constant. The rule that told it to do that was absolute.
215
+
216
+ It is scoped now: run the check that covers what you changed, run the one the
217
+ task named, and do not re-run a check that already passed. Do not install
218
+ anything the task did not ask for. And "locate before you read" is
219
+ conditional - when the question already names the file, read it.
220
+
221
+ Measured over 20 runs after the change, against 60 before:
222
+
223
+ | per run | before | after |
224
+ |---|---|---|
225
+ | `npm test` calls | 2.2 | **0.9** |
226
+ | `npm install` calls | 0.4 | **0** |
227
+
228
+ ### Fixed, again
229
+
230
+ A review of this release found six more, and fixing them turned up two of the
231
+ same class.
232
+
233
+ **The project-root fix was incomplete.** Two call sites still read the
234
+ worktree: the effort hook looked for the spent-route file under it, and the
235
+ router wrote the spent-route file under it. So the router recorded a spent
236
+ route where the effort hook was not looking, and a lane the router had
237
+ already given up on was put back onto it.
238
+
239
+ **The documented profile override had never worked.**
240
+ `OPENCODE_HARNESS_PROFILE` is a profile ID, and it was being checked with
241
+ `existsSync`, like the marker file beside it. It never matched a file, so
242
+ the override that the docs and the measurement scripts both rely on was
243
+ inert - and the measurement it gates reported 0% removed and failed.
244
+
245
+ **The measurement depended on the developer's local state.** The pruner
246
+ finds its profile from `.space/active` in the project, and that file is
247
+ gitignored, so on a fresh clone there is none, the pruner does nothing, and
248
+ the gate says the kit removes nothing. The environment override is set for
249
+ the measurement now, and the measurement runs against a scratch directory
250
+ rather than writing into the repository it is measuring.
251
+
252
+ **A gate that could be switched off by a typo.** `--min-saved abc` became
253
+ NaN, and `NaN` compares false against everything, so the gate never failed.
254
+ The script uses the shared flag parser, which throws.
255
+
256
+ **The A/B stopped the whole experiment twice, both times wrongly.** Once on a
257
+ scratch file in the temp directory, which is the one place GLOBAL.md tells an
258
+ agent to put scratch files and the one place the guard's scope rule allows;
259
+ and once on an agent that grepped the filesystem root. A READ outside the
260
+ subject is wasteful and sometimes a privacy leak, but it damages nothing - the
261
+ pilot that mattered WROTE to the real repository. A write outside the target
262
+ stops the experiment; a read outside it is counted and reported.
263
+
264
+ **A citation to the suite's own file was accepted as evidence.** The self
265
+ suite's subject is a copy of this repository, so the answer key is inside
266
+ it, and an agent that greps for the symbol could cite the suite and pass
267
+ without reading the source. The suite files are excluded from the citations.
268
+
269
+ **The failover threw away the utility route.** `applySessionModel` set
270
+ `small_model` to the fallback profile's session model, so the fleet's cheap
271
+ route was replaced exactly when the fleet failed over - and `verify` then
272
+ called the difference between the file on disk and the running session a
273
+ drift.
274
+
275
+ Also: the python fallback ran every query twice, the CLI check spawned a
276
+ process per query, the database path was not URI-escaped, `--since` was
277
+ ignored as the last argument, `--interval abc` became a 1ms loop, a failed
278
+ catalogue fetch in watch mode announced every missing model as "is back", a
279
+ non-list `plugin` value was replaced instead of kept, the `why` in
280
+ fleet-all's utility route claimed the model was in every chain when it is in
281
+ seven of ten, and the install-wide measurement could report a 100% context
282
+ improvement from a NULL average.
283
+
284
+ ### Changed
285
+
286
+ - **The install-wide measurement is an observation, and there is a controlled
287
+ one.** `measure:check` compared the work before the kit landed with the work
288
+ after, and gated on the context per request. On this install it said the
289
+ context per request went UP by 16%, with twice as many reads and twice as
290
+ many messages in the second window - a number about the jobs, not about the
291
+ kit. The after-window was also "everything since the install", so the
292
+ comparison was a different one every time the script ran.
293
+
294
+ The windows are fixed and equal around the split now, and the report says
295
+ how long they were. The context drop is still reported and still gated, but
296
+ only when the two windows are comparable: when one side did twice the work
297
+ of the other, the number is skipped with a note that says why, because a gate
298
+ that fails for reasons nobody can act on is a gate people switch off.
299
+
300
+ The gate uses the number that IS comparable. opencode stores the message
301
+ parts UNPRUNED - the pruner rewrites the outgoing request, not the history
302
+ - so the stored parts of a real session are exactly what the pruner was
303
+ given, and replaying the real pruner over them measures exactly what it
304
+ removed, on the same messages with and without. On this install: **80.5% of
305
+ the context across 60 real sessions**, 6,608 parts stubbed - the number
306
+ `npm run measure:prune` prints.
307
+
308
+ The measurement scripts also run without the sqlite3 CLI now. python3's
309
+ built-in sqlite3 module does the same job, and python3 is on more machines
310
+ than the CLI is.
311
+
312
+ - **`small_model` is a cheaper route.** It is what opencode uses for its short
313
+ auxiliary calls - a title, a one-line summary - and it was the session model,
314
+ so every one of those calls paid the session model's REASONING. Measured on
315
+ this tier, a title costs 49 reasoning tokens on deepseek and 332 on
316
+ nemotron-3.5-super-vl, and deepseek answers in 7.2s against 22.7s for glm. A
317
+ fleet names a cheaper route now and the generated file says which one; a
318
+ single-model profile keeps its one route, because "one route, and it is the
319
+ one you asked for" is a promise those profiles make.
320
+
321
+ - **The safe fleet honours every tier request it makes.** The tier is worth
322
+ 17x on reasoning tokens - deepseek spends 320 a message at `max` and 19 at
323
+ `low` - so a lane that asks for a tier and lands on a model that publishes
324
+ none runs at that model's natural rate instead. One lane was genuinely
325
+ expensive: `sweep` is a bulk mechanical search, so reasoning on it is pure
326
+ waste, and it ran on a model that reasons 420 tokens a message at 16.6s.
327
+ No zero-retention model publishes a `none` level, so the lane could not even
328
+ turn reasoning off when it asked to. It runs on deepseek now: 19 reasoning
329
+ tokens at 3.4s.
330
+
331
+ The three lanes in the unrestricted fleet that cannot get their tier turn
332
+ out to be the three CHEAPEST reasoners on the tier - 116, 59 and 18 tokens a
333
+ message, against 289 for `build` and 198 for `plan`, which keep their tier
334
+ on purpose. The tier is the lever, and it is not where the tokens are. The
335
+ generated config says so where it happens, in the form
336
+ `locate=unset(asked low)`.
337
+
338
+ - `npm run check` is the offline gate — build, typecheck, tests — and is green
339
+ on a checkout that has never seen an OpenCode login. The live half moved to
340
+ `npm run check:install`, which needs a machine that has run OpenCode. The
341
+ old claim that `npm run check` had to pass on an untouched checkout was
342
+ false, and it was the reason the whole gate was skipped.
343
+ - `profiles/schema.json` no longer accepts `roleModels.<lane>.levels`. The
344
+ field was in the schema, validated as an array, and then read by nothing, so
345
+ a profile author could fill it with a lie and get no error.
346
+ - `oc-harness show` now prints what the guard blocks and why. It was the one
347
+ part of the kit a user could not inspect at all.
348
+ - The config directory path uses `homedir()` in all three places that build
349
+ it. One of them used `process.env.HOME`, which with HOME unset produced a
350
+ RELATIVE path, so `use` wrote to one place and the reader read another.
351
+ - `retention.yml` is cached like the lane table next to it. It was re-read and
352
+ re-parsed for every candidate of every lane.
353
+ - The severity table ranked `unknown` as milder than `may-train`, while
354
+ `retentionAllowed` refuses `unknown` outright. The two contradicted each
355
+ other, and the table is the one that decides what a profile is labelled.
356
+ - The `lanes` header printed the raw record count and called it "lane
357
+ request(s)". A request is one assistant message, and a message is written
358
+ twice, so the headline said twice what the table under it said.
359
+ - The router checked the health file for a profile ID where the file is keyed
360
+ by MODEL. For the ten single-model profiles the two are the same string and
361
+ the check worked by accident; for the two fleets it could never fire.
362
+ - `harness.fallback` in the generated config was a copy of the first id of a
363
+ queue the router reads straight from the profile file, in a key nothing
364
+ reads. `fallbackProfileId` was a parameter no caller ever supplied.
365
+ - The README quoted two different sets of numbers for the same measurement.
366
+ There is one set now, and it is the one in `docs/RESEARCH.md` and
367
+ `eval/RESULTS.md`.
368
+ - Both `.space/telemetry.jsonl` and `.space/prune.jsonl` are written in the
369
+ project. The prune log used to land in the process's working directory,
370
+ which is often your home directory.
371
+ ### Added
372
+
373
+ - **`show --json` names the profile's own notes `notes`** and the compiler's
374
+ adjustments `adjustments`, because the human report calls them different
375
+ things and the JSON should not make one name mean two.
376
+
377
+ - **`doctor --json` and `show --json`.** "Is the install healthy" and "what
378
+ does this profile resolve to" are questions a CI job and a dashboard ask,
379
+ and the answers were tables of padded columns with colour codes in them.
380
+ Both now print the machine-readable form and nothing else - not the human
381
+ report first and the JSON after it, which is not machine-readable. An
382
+ error in JSON mode is a JSON object, so a script that pipes the output
383
+ into a parser gets something it can read either way. `lanes --json` used
384
+ the key `lanes` for the USAGE of each lane, and `show --json` uses it for
385
+ the lane TABLE; the usage one is now `usage`, so one name does not mean
386
+ two shapes.
387
+
388
+ - **`docs/EXIT-CODES.md`**, because a command that can fail should say how,
389
+ and "non-zero" does not tell a script whether the work landed or there was
390
+ no task file. Four codes for the loop, two for the doctor, and the rule
391
+ that a measurement which could not be compared is reported and NOT gated.
392
+
393
+ - **`MIGRATION.md` covers 0.3 to 0.4.** The upgrade asks something of the
394
+ user - the guard now refuses in code what it used to only suggest, and the
395
+ pre-commit hook runs the project's own tests instead of the kit's - so it
396
+ is written down, with the command to check it and the command to undo it.
397
+
398
+ - **`oc-harness cost`** — what the kit is doing on THIS project, in numbers.
399
+ The kit's claim is a trade: a fixed overhead on every request, and the
400
+ context it prunes back. Both numbers were already being written — the prune
401
+ log records what each pass removed, and the telemetry records what each lane
402
+ cost — and nothing read them together. On the one measured here: 83.7
403
+ million characters removed over 272 prune passes, 48,122 tool outputs
404
+ stubbed, and the lanes sorted by what they cost.
405
+
406
+ - **`oc-harness uninstall`.** The install SYMLINKS the plugin, the prompts,
407
+ the skills and GLOBAL.md out of the kit's directory, so `npm uninstall` left
408
+ four dangling symlinks and opencode then loaded neither the rules nor the
409
+ guard — and said nothing about it.
410
+
411
+ - **`oc-harness refresh --all`**, which re-derives every shipped profile
412
+ against the live catalogue, and updates `verified`, which is the field that
413
+ says when a profile was last checked. Leaving it stale made it a lie.
414
+
415
+ - **`oc-harness upgrade`**, which checks the registry for a newer kit,
416
+ refreshes the profiles, and writes the config again. Three things go stale
417
+ on their own and all three were invisible.
418
+
419
+ - **`oc-harness backups` and `oc-harness restore`.** The single `.bak` was
420
+ overwritten on the next `use` and carried no profile id, so "go back to
421
+ what I had" was not something a person could do at 2am. The config also
422
+ records which kit wrote it.
423
+
424
+ - **Type declarations.** The package's `types` pointed at `src/index.ts` — a
425
+ TypeScript SOURCE file — and the tsconfig had `noEmit`, so no declarations
426
+ were ever produced. A consumer type-checked the kit's source out of
427
+ node_modules.
428
+
429
+ - **Tests for the pieces the suite never reached**: the python fallback, the
430
+ project root for a project that is not a git repository, the escape rule,
431
+ the flag parser, and the measurement scripts themselves — the gate that
432
+ `npm run measure:check` depends on had no test at all.
433
+
434
+ - **The demo is back on the front page.** `docs/demo/pitch.mp4` and
435
+ `docs/demo-loop.gif` are in the README again, and `video/` still holds the
436
+ take script, the voiceover and the recording driver. The film was recorded
437
+ against an earlier version of the kit, and the README now says so next to
438
+ it rather than leaving a front page that quietly misstates the project's own
439
+ numbers - which is the reason 0.4.3 removed it in the first place, and the
440
+ reason it should not have been removed without putting the film back.
441
+
442
+ - **`OPENCODE_HARNESS_CATALOG_URL` overrides where the live model catalogue
443
+ comes from.** A mirror is one use; a local fixture is another, and a test
444
+ that reached models.dev would fail on a machine with no network. The `file://`
445
+ spelling is read directly, so a fixture needs no external binary.
446
+
447
+ - **`OC_HARNESS_NO_HOOK` skips the pre-commit hook.** A test that writes an
448
+ executable git hook into the repository it is testing is a test that changes
449
+ the repository.
450
+
451
+ - **`oc-harness use --memory` turns on a memory plugin.** The kit does not
452
+ implement memory, and it should not: a local vector store with embeddings, a
453
+ web UI, auto-capture and a user profile is a large thing to build badly, and
454
+ `opencode-mem` already is one. The entry is ADDED to the plugin list, so the
455
+ user's own plugins survive and a config that already has it does not grow a
456
+ second copy.
457
+
458
+ - **`oc-harness doctor --check-models` and `--watch`.** The question a
459
+ free-tier user actually has to ask is "is one of my routes gone", and it is
460
+ not the same question as "has a profile drifted". `--check-models` answers it
461
+ in one run with a non-zero exit, which is the shape a cron wants; `--watch`
462
+ keeps running and reports only what CHANGED.
463
+
464
+ - **`eval/measure-prune.mjs`.** The attributable measurement of the pruner,
465
+ described above.
466
+
467
+ - **The A/B suite is reproducible, and it has been run.** The old one ran
468
+ against `lablab-harness@8a674f5`, which is not public. A result nobody can
469
+ check is not a result. The new suite runs against the kit's own tree at a
470
+ pinned commit, so `git clone && git checkout <sha>` reproduces it exactly.
471
+
472
+ Ten reps per arm, 120 agent runs, and the result is not flattering:
473
+
474
+ | metric (median) | kit on | kit off |
475
+ |---|---|---|
476
+ | **correct** | **52/60** | **59/59** |
477
+ | tool calls | 11.5 | 3 |
478
+ | input tokens | 51,449 | 4,551 |
479
+ | final context | 39,603 | 10,855 |
480
+ | wall seconds | 43.6 | 32.5 |
481
+ | paths written outside the target | 0 | 0 |
482
+
483
+ **On a task that fits in one read, the kit is a cost.** Its instructions
484
+ cost eleven times the input tokens and four times the tool calls, the run
485
+ takes a third longer, and the answers are worse. The rule that says *locate
486
+ before you read* is a waste on a one-file question, and the tool counts show
487
+ it. The two edit tasks are a tie at 10/10, and the guard stopped no writes
488
+ in either arm.
489
+
490
+ The value the kit does have is in long sessions - the pruner removes 80.5%
491
+ of the context across real ones - and in process, which a grader that
492
+ counts correct answers cannot see. Whether it is worth it depends on
493
+ whether the work is long. **A repo of one-file lookups should not install
494
+ this.**
495
+
496
+ The full table, and what it does not establish, is in
497
+ `eval/RESULTS.md`.
498
+
499
+ - **`oc-harness refresh --all`** re-derives every shipped profile against the
500
+ live catalogue, and reports which models are gone.
501
+ - **`oc-harness doctor --check-models` and `--watch`** record a withdrawal in
502
+ `~/.config/opencode/withdrawn.jsonl`, with the profiles that used it. A model
503
+ is withdrawn without a rename and without a notice, so the only way to know
504
+ WHEN a lane started failing is to have been looking.
505
+ - **`oc-harness upgrade`** checks the registry for a newer kit, refreshes the
506
+ profiles, and writes the config again. Three things go stale on their own -
507
+ the kit, the profiles and the config - and all three were invisible.
508
+ - **A scheduled workflow** (`.github/workflows/measure.yml`), so the numbers
509
+ are kept rather than measured once. The offline half runs on every push;
510
+ the A/B runs nightly, because it costs model quota and takes about an hour.
511
+ - **`OPENCODE_HARNESS_PROFILES`** points the kit at a different profile
512
+ directory. A test that refreshes a profile would otherwise rewrite the kit's
513
+ own data - which is what happened the first time this was tried.
514
+ - **A flag appended to an npm script now overrides the script's own**:
515
+ `npm run measure:prune -- --sessions 1` means one session, not sixty.
516
+
517
+ - **Tests for the pieces the suite never reached**: the python fallback, the
518
+ project root for a project that is not a git repository, the escape rule,
519
+ and the flag parser.
520
+
521
+ ### Security
522
+
523
+ - `eval/prepare.mjs` could delete a directory tree. `--name` was joined onto
524
+ the scratch root without being checked and the result removed with
525
+ `rmSync(..., { recursive: true })`, so `--name ../victim` deleted the
526
+ victim's tree. The label is a name and not a path, and the resolved path is
527
+ checked a second time against the scratch root.
528
+ - `eval/prepare.mjs` could run a command. The archive step was one `sh -c`
529
+ string and the paths were quoted with `JSON.stringify`, which is not shell
530
+ quoting — a path containing `$(...)` was executed by the shell. `git` and
531
+ `tar` are called with argument arrays now, and the tar goes through a pipe.
532
+
533
+ ## [0.4.3] — 2026-10-20
534
+
535
+ ### Removed
536
+
537
+ - **The video pipeline and the published demo.** The `video/` directory held
538
+ the take script, the voiceover, the recording driver and the superseded
539
+ dubbing pipeline; `docs/demo/pitch.mp4` and `docs/demo-loop.gif` held the
540
+ published film. This repository is a harness kit, and it is code.
541
+
542
+ The film was also stale: it said "seven profiles" and "one hundred and
543
+ sixty five tests", and the kit now ships twelve profiles and two hundred
544
+ and ten tests. A front page that misstates the project's own numbers is
545
+ worse than no front page, so it goes rather than gets quietly outdated
546
+ again.
16
547
 
17
548
  ## [0.4.2] — 2026-10-20
18
549
 
@@ -42,20 +573,11 @@ is not happening.
42
573
  that can be opened in the JSON body. The list asked for JSON and worked; the
43
574
  create did not, and failed with a message that said nothing about why. Every
44
575
  npm trust call asks for JSON now.
45
- - **Each npm command gets its own two-factor challenge.** The code a browser
46
- flow returns is valid for the challenge that asked for it and for nothing
47
- else, so carrying one from one command to the next made the second command
48
- send a code that was already spent - and the registry answered that with a
49
- different challenge, one that wants an authenticator code rather than a
50
- browser. The failure looked like "npm wants something else now" and nothing
51
- like the cause.
52
- - **The URLs come from the response body, not from a guess.** The done URL
53
- was rebuilt from the id, which sent a real request to the registry with a
54
- made-up id; the registry answered "forbidden", which is not a token, so the
55
- wait ran its full length and then reported that nobody had answered.
56
- - **`OC_TRUST_NO_BROWSER` turns the browser off.** A test that opens a
57
- browser on the machine it runs on is a test that breaks whatever the
58
- person at the keyboard was doing.
576
+ - The per-command two-factor challenge, the done-URL guess and
577
+ `OC_TRUST_NO_BROWSER` were all fixed in 0.3.9 and are listed there. They
578
+ appear here once more because this release was cut from a branch that
579
+ predated 0.3.9's changelog entry, and a reader of the two releases was
580
+ left with the impression that the same three fixes happened twice.
59
581
 
60
582
  ### The release path, end to end
61
583
 
@@ -167,8 +689,9 @@ The publish is now automatic and holds no secret:
167
689
  short-lived publish token, scoped to the workflow that asked for it. There
168
690
  is no long-lived token in the repository to leak, rotate or expire - and npm
169
691
  is revoking the legacy automation tokens anyway.
170
- - The job installs npm 11.5.1 or later, which is what OIDC needs and what
171
- node 22 does not ship.
692
+ - The job runs on node 24, whose bundled npm already speaks trusted
693
+ publishing. An `npm install -g npm@latest` self-upgrade also works, and is
694
+ documented elsewhere as intermittently corrupting the install.
172
695
  - The registry is configured by hand instead of through `setup-node`'s
173
696
  `registry-url`, because that helper always writes an auth line, and an
174
697
  empty auth line makes npm believe it is authenticated and skip the OIDC
@@ -381,7 +904,7 @@ the tool-output caps, and the `scout` → `locate` agent rename.
381
904
  `git log --grep push` and `echo push` are not. A heredoc body is data, not
382
905
  commands.
383
906
  - **`tool_output` was pinned at 200 lines / 8 KB**, about 6× tighter than
384
- OpenCode's default. It truncated 415 of 2,966 `read` calls and 418 calls
907
+ OpenCode's default. It truncated 415 of 2,966 `read` calls and 423 calls
385
908
  were issued with an `offset` to fetch the rest: the truncation *causes* the
386
909
  re-read, one for one, at a measured 17.3 s median per request. The cap is
387
910
  now the default, and pruning rather than truncation bounds the context.
@@ -433,7 +956,9 @@ silent in the worst way:
433
956
 
434
957
  ---
435
958
 
436
- [Unreleased]: https://github.com/pham-canh/space-harness/compare/v0.4.2...HEAD
959
+ [Unreleased]: https://github.com/pham-canh/space-harness/compare/v0.4.3...HEAD
960
+ [0.4.4]: https://github.com/pham-canh/space-harness/compare/v0.4.3...v0.4.4
961
+ [0.4.3]: https://github.com/pham-canh/space-harness/compare/v0.4.2...v0.4.3
437
962
  [0.4.2]: https://github.com/pham-canh/space-harness/compare/v0.4.1...v0.4.2
438
963
  [0.4.1]: https://github.com/pham-canh/space-harness/compare/v0.4.0...v0.4.1
439
964
  [0.4.0]: https://github.com/pham-canh/space-harness/compare/v0.3.9...v0.4.0