opencode-harness-kit 0.5.1 → 0.5.4

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/CHANGELOG.md CHANGED
@@ -12,6 +12,109 @@ is not happening.
12
12
 
13
13
  ---
14
14
 
15
+ ## [0.5.4] — 2026-10-21
16
+
17
+ Published. The tag is what cut it: `git tag v0.5.4 && git push origin main
18
+ v0.5.4`, and the publish job watches for exactly that.
19
+
20
+ A patch, and the patch is the publish path. 0.5.3 was the first release to
21
+ reach the registry since the rename, and it reached it through trusted
22
+ publishing with a signed provenance statement - so the path works, and this
23
+ release is the one that says so where a reader will look for it.
24
+
25
+ ### Changed
26
+
27
+ - **The npm trusted publisher is configured for
28
+ `pham-canh/opencode-harness-kit`**: `ci.yml`, no environment, publish and
29
+ stage. It is the one thing that was missing when 0.5.2 was refused, and it
30
+ is set now. The re-point is an account change on npm, so it needed the
31
+ account's second factor - and a rename is not a thing npm does for you.
32
+
33
+ - **The publish token secret is gone.** The job can publish with a token
34
+ when there is one, and the secret was set while 0.5.3 was cut, because a
35
+ release that cannot publish is not a release. It is removed now that the
36
+ OIDC path is proven: a long-lived token in the repository is a secret to
37
+ rotate for a path that is not being used, and npm is restricting the
38
+ tokens that bypass two-factor authentication for direct publishing in
39
+ January 2027, so the token path was on its way out anyway.
40
+
41
+ ## [0.5.3] — 2026-10-21
42
+
43
+ Published. The tag is what cut it: `git tag v0.5.3 && git push origin main
44
+ v0.5.3`, and the publish job watches for exactly that.
45
+
46
+ This is the release that carries the rename, because 0.5.2 - the release the
47
+ rename was written for - never reached the registry.
48
+
49
+ ### Changed
50
+
51
+ - **The repository is now `pham-canh/opencode-harness-kit`**, renamed from
52
+ `pham-canh/space-harness`. The npm package has been
53
+ `opencode-harness-kit` since 0.4.0, and the README's title with it; the
54
+ repository was the last one still carrying the old name, and a reader who
55
+ installed the package and followed the link landed somewhere that did not
56
+ look like the thing they had installed. GitHub redirects the old URL, so
57
+ the links that already exist keep working.
58
+
59
+ The npm page showed the old name until this release, and the reason is
60
+ worth writing down, because nothing about the rename explains it: the page
61
+ reads the `repository` and `homepage` out of the manifest of the LATEST
62
+ PUBLISHED version, and the latest published version was 0.5.1, whose
63
+ manifest was written before the rename. The manifest names the new
64
+ repository now, and this is the release that carries it.
65
+
66
+ ### Fixed
67
+
68
+ - **The publish job now checks the OIDC claim against `package.json` before
69
+ it publishes.** npm matches the trusted publisher's repository against the
70
+ claim, and the claim names the repository the workflow ran in - so a
71
+ renamed repository stops publishing, with an exchange error that does not
72
+ name the field that was wrong. The gate says which side is which and
73
+ prints the command that fixes it.
74
+
75
+ - **The installer no longer says "installed" when it replaced nothing.**
76
+ `existsSync` follows a symlink, so a link whose target no longer exists
77
+ reads as "not there" - and the branch that replaces what the installer put
78
+ there was never reached. `symlinkSync` then threw EEXIST over the link
79
+ that was already there, the warning was printed where nobody reads it, and
80
+ the install carried on. After the rename that is exactly the state a kit
81
+ installed from the old path was in: the skills, the prompts and GLOBAL.md
82
+ kept pointing at a directory that had moved, and `verify` then found
83
+ seventeen broken paths. `lstat` does not follow the link, and it is what
84
+ decides whether there is something to replace.
85
+
86
+ ### Removed
87
+
88
+ - **The coverage report is no longer committed.** 131 files of it, four
89
+ megabytes, changing on every run, and nobody reads a diff of it. It is
90
+ ignored now and produced on demand.
91
+
92
+ - **The two shell scripts are gone from the repository**: the demo setup
93
+ helper, which pointed at copies of two tools that live one directory up
94
+ with their own history, and the take script that filmed the published
95
+ demo, which is a shell file with one machine's paths in it and goes stale
96
+ the moment the thing it films changes. The recording guide is on the
97
+ machine that records.
98
+
99
+ ## [0.5.2] — 2026-10-21
100
+
101
+ **Not published.** The tag was cut and pushed, the publish job ran, and npm
102
+ refused the publish - so there is no 0.5.2 on the registry. The changes
103
+ written for it shipped in 0.5.3.
104
+
105
+ What npm said was `OIDC token exchange error - package not found`, which
106
+ reads like a missing package. The package was there. The trusted publisher
107
+ was not: npm reports no trusted publisher configured for this package, and
108
+ the repository had been renamed, so a publisher that named the old
109
+ repository would not have matched the claim the runner presents either. The
110
+ package's name is in the error, and the field that was wrong is not.
111
+
112
+ The entry this release was going to carry said that "the npm trusted
113
+ publisher was re-pointed at the new name". It had not been. The re-point is
114
+ an account change on npm and it needs the account's second factor, and
115
+ nothing had done it. A changelog that says a thing has been done that has
116
+ not been done is worse than one that says nothing.
117
+
15
118
  ## [0.5.1] — 2026-10-21
16
119
 
17
120
  Published. The tag is what cut it: `git tag v0.5.1 && git push origin main
@@ -1130,20 +1233,27 @@ silent in the worst way:
1130
1233
 
1131
1234
  ---
1132
1235
 
1133
- [Unreleased]: https://github.com/pham-canh/space-harness/compare/v0.4.3...HEAD
1134
- [0.4.4]: https://github.com/pham-canh/space-harness/compare/v0.4.3...v0.4.4
1135
- [0.4.3]: https://github.com/pham-canh/space-harness/compare/v0.4.2...v0.4.3
1136
- [0.4.2]: https://github.com/pham-canh/space-harness/compare/v0.4.1...v0.4.2
1137
- [0.4.1]: https://github.com/pham-canh/space-harness/compare/v0.4.0...v0.4.1
1138
- [0.4.0]: https://github.com/pham-canh/space-harness/compare/v0.3.9...v0.4.0
1139
- [0.3.9]: https://github.com/pham-canh/space-harness/compare/v0.3.8...v0.3.9
1140
- [0.3.8]: https://github.com/pham-canh/space-harness/compare/v0.3.7...v0.3.8
1141
- [0.3.7]: https://github.com/pham-canh/space-harness/compare/v0.3.6...v0.3.7
1142
- [0.3.6]: https://github.com/pham-canh/space-harness/compare/v0.3.5...v0.3.6
1143
- [0.3.5]: https://github.com/pham-canh/space-harness/compare/v0.3.4...v0.3.5
1144
- [0.3.4]: https://github.com/pham-canh/space-harness/compare/v0.3.1...v0.3.4
1145
- [0.3.1]: https://github.com/pham-canh/space-harness/compare/v0.3.0...v0.3.1
1146
- [0.3.0]: https://github.com/pham-canh/space-harness/compare/v0.2.1...v0.3.0
1147
- [0.2.1]: https://github.com/pham-canh/space-harness/compare/v0.2.0...v0.2.1
1148
- [0.2.0]: https://github.com/pham-canh/space-harness/compare/v0.1.0...v0.2.0
1149
- [0.1.0]: https://github.com/pham-canh/space-harness/releases/tag/v0.1.0
1236
+ [Unreleased]: https://github.com/pham-canh/opencode-harness-kit/compare/v0.5.4...HEAD
1237
+ [0.5.4]: https://github.com/pham-canh/opencode-harness-kit/compare/v0.5.3...v0.5.4
1238
+ [0.5.3]: https://github.com/pham-canh/opencode-harness-kit/compare/v0.5.2...v0.5.3
1239
+ [0.5.2]: https://github.com/pham-canh/opencode-harness-kit/compare/v0.5.1...v0.5.2
1240
+ [0.5.1]: https://github.com/pham-canh/opencode-harness-kit/compare/v0.5.0...v0.5.1
1241
+ [0.5.0]: https://github.com/pham-canh/opencode-harness-kit/compare/v0.4.6...v0.5.0
1242
+ [0.4.6]: https://github.com/pham-canh/opencode-harness-kit/compare/v0.4.5...v0.4.6
1243
+ [0.4.5]: https://github.com/pham-canh/opencode-harness-kit/compare/v0.4.4...v0.4.5
1244
+ [0.4.4]: https://github.com/pham-canh/opencode-harness-kit/compare/v0.4.3...v0.4.4
1245
+ [0.4.3]: https://github.com/pham-canh/opencode-harness-kit/compare/v0.4.2...v0.4.3
1246
+ [0.4.2]: https://github.com/pham-canh/opencode-harness-kit/compare/v0.4.1...v0.4.2
1247
+ [0.4.1]: https://github.com/pham-canh/opencode-harness-kit/compare/v0.4.0...v0.4.1
1248
+ [0.4.0]: https://github.com/pham-canh/opencode-harness-kit/compare/v0.3.9...v0.4.0
1249
+ [0.3.9]: https://github.com/pham-canh/opencode-harness-kit/compare/v0.3.8...v0.3.9
1250
+ [0.3.8]: https://github.com/pham-canh/opencode-harness-kit/compare/v0.3.7...v0.3.8
1251
+ [0.3.7]: https://github.com/pham-canh/opencode-harness-kit/compare/v0.3.6...v0.3.7
1252
+ [0.3.6]: https://github.com/pham-canh/opencode-harness-kit/compare/v0.3.5...v0.3.6
1253
+ [0.3.5]: https://github.com/pham-canh/opencode-harness-kit/compare/v0.3.4...v0.3.5
1254
+ [0.3.4]: https://github.com/pham-canh/opencode-harness-kit/compare/v0.3.1...v0.3.4
1255
+ [0.3.1]: https://github.com/pham-canh/opencode-harness-kit/compare/v0.3.0...v0.3.1
1256
+ [0.3.0]: https://github.com/pham-canh/opencode-harness-kit/compare/v0.2.1...v0.3.0
1257
+ [0.2.1]: https://github.com/pham-canh/opencode-harness-kit/compare/v0.2.0...v0.2.1
1258
+ [0.2.0]: https://github.com/pham-canh/opencode-harness-kit/compare/v0.1.0...v0.2.0
1259
+ [0.1.0]: https://github.com/pham-canh/opencode-harness-kit/releases/tag/v0.1.0
package/CONTRIBUTING.md CHANGED
@@ -139,7 +139,7 @@ repository secret to a granular token scoped to `opencode-harness-kit` with
139
139
  if it is absent the job uses OIDC.
140
140
 
141
141
  ```bash
142
- gh secret set NPM_TOKEN --repo pham-canh/space-harness
142
+ gh secret set NPM_TOKEN --repo pham-canh/opencode-harness-kit
143
143
  ```
144
144
 
145
145
  > **`gh secret set` takes the secret NAME as its first argument**, and the
@@ -213,7 +213,7 @@ To set it without the script, the command is:
213
213
  ```bash
214
214
  npm trust github opencode-harness-kit \
215
215
  --file ci.yml \
216
- --repo pham-canh/space-harness \
216
+ --repo pham-canh/opencode-harness-kit \
217
217
  --allow-publish --allow-stage-publish -y
218
218
  ```
219
219
 
@@ -222,7 +222,7 @@ The configuration it writes is:
222
222
  | field | value |
223
223
  |---|---|
224
224
  | Organization or user | `pham-canh` |
225
- | Repository | `space-harness` |
225
+ | Repository | `opencode-harness-kit` |
226
226
  | Workflow filename | `ci.yml` (the filename only, not a path) |
227
227
  | Environment name | **empty** |
228
228
  | Allowed actions | `npm publish` and `npm stage publish` |
@@ -236,7 +236,7 @@ The workflow is the one in `.github/workflows/ci.yml`, and the claims it
236
236
  presents are printed by the publish job before it tries:
237
237
 
238
238
  ```
239
- repository pham-canh/space-harness
239
+ repository pham-canh/opencode-harness-kit
240
240
  workflow file ci.yml
241
241
  environment (absent) - the trusted publisher must name NO environment
242
242
  owner pham-canh
package/README.md CHANGED
@@ -1,6 +1,6 @@
1
1
  # opencode-harness-kit
2
2
 
3
- [![CI](https://github.com/pham-canh/space-harness/actions/workflows/ci.yml/badge.svg)](https://github.com/pham-canh/space-harness/actions/workflows/ci.yml)
3
+ [![CI](https://github.com/pham-canh/opencode-harness-kit/actions/workflows/ci.yml/badge.svg)](https://github.com/pham-canh/opencode-harness-kit/actions/workflows/ci.yml)
4
4
  [![npm](https://img.shields.io/npm/v/opencode-harness-kit.svg)](https://www.npmjs.com/package/opencode-harness-kit)
5
5
 
6
6
  <p align="center">
@@ -8,9 +8,9 @@
8
8
  </p>
9
9
 
10
10
  *Four minutes forty-seven seconds, recorded live from a real terminal and
11
- narrated. The take script and the voiceover are in [`video/`](video/), and
12
- rebuilding it is documented there — including the two mistakes that cost the
13
- most time, so the next person does not repeat them.*
11
+ narrated — every command on screen is typed and executed while the camera
12
+ runs. The voiceover is in [`video/`](video/), along with what it took to
13
+ record it and the two mistakes that cost the most time.*
14
14
 
15
15
  *It was recorded against an earlier version of the kit: the numbers it names
16
16
  are the ones from then, and the ones below are the ones now.*
@@ -31,7 +31,7 @@ your side — `npx` only downloads the published kit. Prefer git? The same three
31
31
  steps from a checkout (see [Install](#install)):
32
32
 
33
33
  ```bash
34
- git clone https://github.com/pham-canh/space-harness.git
34
+ git clone https://github.com/pham-canh/opencode-harness-kit.git
35
35
  ./bin/oc-harness.mjs init
36
36
  ```
37
37
 
@@ -67,6 +67,33 @@ src/ ← no model name anywhere in here
67
67
  Same plugin source, twelve shipped profiles, twelve different behaviours. The
68
68
  test suite asserts that rather than taking it on faith.
69
69
 
70
+ ### How a request travels
71
+
72
+ ```mermaid
73
+ flowchart LR
74
+ P[profile JSON<br/>model, effort, fallback, retention] --> C[oc-harness use<br/>compiles the profile]
75
+ C --> CFG[~/.config/opencode/opencode.jsonc<br/>agent.&lt;lane&gt;.model]
76
+ CFG --> PL[the plugin: guard, prune, router, telemetry]
77
+ PL --> LANES[10 lanes<br/>each on the model built for it]
78
+ LANES --> PR[prune: old tool output<br/>becomes a stub]
79
+ LANES --> GD[guard: denied commands<br/>and writes outside the project]
80
+ LANES --> RT[router: a spent route<br/>moves to the next rung]
81
+ LANES --> TL[telemetry: what each lane<br/>actually cost]
82
+ ```
83
+
84
+ Four things happen to a request, and they are four separate files:
85
+
86
+ | | |
87
+ |---|---|
88
+ | `src/guard.ts` | refuses the commands the kit never allows, and any write outside the project |
89
+ | `src/prune.ts` | replaces old tool output with a one-line stub, so the window holds the work and not the noise |
90
+ | `src/router.ts` | when a route is spent, moves the lane to the next rung of its own chain |
91
+ | `src/telemetry.ts` | records what each lane actually cost, so `oc-harness lanes` can say which ones earn their place |
92
+
93
+ The model for a lane is in the **config**, not in a prompt and not in a file
94
+ the agent reads. That is the whole design: the answers are data, and a data
95
+ edit is reviewable.
96
+
70
97
  ## A fleet, not a model
71
98
 
72
99
  The default profile is a **fleet**: ten jobs, ten free models, one session.
@@ -690,8 +717,8 @@ Without registry access, the checkout path works identically:
690
717
 
691
718
  ```bash
692
719
  pkg install nodejs git
693
- git clone https://github.com/pham-canh/space-harness.git
694
- ./space-harness/bin/oc-harness.mjs init
720
+ git clone https://github.com/pham-canh/opencode-harness-kit.git
721
+ ./opencode-harness-kit/bin/oc-harness.mjs init
695
722
  ```
696
723
 
697
724
  The kit writes to `$HOME/.config/opencode` like everywhere else, so there is
@@ -863,6 +890,38 @@ session, so switching happens at the next session, not mid-flight. What a
863
890
  plugin *can* do is move a lane to the next rung of its chain before the
864
891
  session starts.
865
892
 
893
+ ## What is next, and what is not measured yet
894
+
895
+ The claims this project makes are measured, and where a measurement has not
896
+ happened the claim is written down as a gap rather than as a fact. These are
897
+ the open ones.
898
+
899
+ **The pruner's rule was measured on one task.** Keeping the newest copy of
900
+ each distinct tool output instead of a window on recent calls cut the input
901
+ tokens by 69% and the wall time by 60% on the long-task A/B. One task is one
902
+ workload. It needs the same treatment on a repository that is not this one,
903
+ and on a session that runs for hours rather than minutes.
904
+
905
+ **The reviewer's "you never fix" is an instruction, not a boundary.** The
906
+ measurement of it came out even: the kit's reviewer lane and a default agent
907
+ behaved the same on a review task, and a task that says "fix it" overrode
908
+ the lane. The kit's own loop keeps the line by never asking a reviewer to
909
+ fix anything. A guard that cannot be overridden by a task is worth having,
910
+ and that is not built yet.
911
+
912
+ **The nightly A/B needs a provider key on the repository.** The workflow
913
+ exists, it runs, and it is skipped until the key is set — the variable
914
+ `HAS_OPENCODE_KEY` says the secret is there. The numbers in
915
+ [eval/RESULTS.md](eval/RESULTS.md) are the ones taken by hand.
916
+
917
+ **The TypeScript pilot is one public repository.** It is now a real one
918
+ (`typesense-js`, pinned by commit) rather than a private tree, which is what
919
+ made it checkable. Two languages are not a survey.
920
+
921
+ **The lanes are ten because ten models were available.** That is a
922
+ hypothesis about the tier, not a number. `oc-harness lanes` counts the
923
+ requests and says which lanes were never used; cutting them is a data edit.
924
+
866
925
  ## Development
867
926
 
868
927
  ```bash
@@ -907,9 +966,15 @@ and the measurements behind each choice.
907
966
 
908
967
  [`CHANGELOG.md`](CHANGELOG.md) records what each release changed and, for the
909
968
  releases before this one, what was broken in it — the silent failures that a
910
- config could look correct through. [`MIGRATION.md`](MIGRATION.md) covers
911
- 0.2.x → 0.3.0: the profile ids changed, the config layout changed, and the
912
- telemetry file now means something different.
969
+ config could look correct through. [`MIGRATION.md`](MIGRATION.md) covers the
970
+ two upgrades that ask something of you: 0.2.x → 0.3.0 (the profile ids
971
+ changed, the config layout changed, the telemetry file now means something
972
+ different) and 0.3.x → 0.4.x (the guard started refusing in code what it used
973
+ to only suggest, and the pre-commit hook started running your project's
974
+ tests instead of the kit's).
975
+
976
+ [`docs/EXIT-CODES.md`](docs/EXIT-CODES.md) lists what every command returns
977
+ when it fails, so a script can tell "the kit says no" from "the kit broke".
913
978
 
914
979
  ## Contributing
915
980
 
package/SECURITY.md CHANGED
@@ -12,7 +12,7 @@
12
12
  ## Reporting a vulnerability
13
13
 
14
14
  Please report privately through GitHub's
15
- [private vulnerability reporting](https://github.com/pham-canh/space-harness/security/advisories/new)
15
+ [private vulnerability reporting](https://github.com/pham-canh/opencode-harness-kit/security/advisories/new)
16
16
  rather than opening a public issue.
17
17
 
18
18
  Include the affected version, a reproduction, and the expected behaviour.
@@ -21,7 +21,7 @@ plan within 7 days.
21
21
 
22
22
  ## Scope
23
23
 
24
- `space-harness` runs agent subprocesses and reads harness config. Reports
24
+ `opencode-harness-kit` runs agent subprocesses and reads harness config. Reports
25
25
  that are in scope include path traversal when resolving a capability
26
26
  profile, command injection through harness config values, and any way a
27
27
  harness writes outside its configured working directory.
@@ -706,19 +706,32 @@ function linkOne(src, dest) {
706
706
  }
707
707
  if (ours) return
708
708
 
709
- if (existsSync(dest)) {
709
+ if (isLink) {
710
+ // A symlink is a symlink whether or not its target exists, and a
711
+ // DANGLING one is still this installer's to replace.
712
+ //
713
+ // `existsSync` follows the link, so it says false for a link whose
714
+ // source has moved - the repository was renamed, or the kit was
715
+ // reinstalled somewhere else. The old code asked `existsSync` whether
716
+ // the destination was taken, got "no", and then tried to create a link
717
+ // over an existing one: EEXIST, printed as a warning nobody reads, and
718
+ // the install reported success. Every skill, prompt and rule kept
719
+ // pointing at a directory that is no longer there, and `verify` found
720
+ // seventeen broken paths afterwards.
721
+ //
722
+ // The test that decides whether to replace something is `lstat`, which
723
+ // does not follow the link.
724
+ rmSync(dest, { force: true })
725
+ } else if (existsSync(dest)) {
710
726
  // Replace only what this installer put there. The old code treated "not
711
727
  // already our link" as "safe to delete" and called `rmSync` with
712
728
  // `recursive: false`, which succeeds on a plain file: a hand-written
713
729
  // agent was deleted and replaced by a symlink without a word. A real
714
730
  // directory is left alone too -- unlinking it is not this installer's
715
731
  // call to make.
716
- if (!isLink) {
717
- const kind = lstatSync(dest).isDirectory() ? "directory" : "file"
718
- console.log(` ${yellow("!")} ${stamp(`left alone: ${dest} is a real ${kind} and was not ours to replace`)}`)
719
- return
720
- }
721
- rmSync(dest, { force: true })
732
+ const kind = lstatSync(dest).isDirectory() ? "directory" : "file"
733
+ console.log(` ${yellow("!")} ${stamp(`left alone: ${dest} is a real ${kind} and was not ours to replace`)}`)
734
+ return
722
735
  }
723
736
 
724
737
  try {
package/eval/RESULTS.md CHANGED
@@ -146,6 +146,115 @@ That is the mechanism the whole kit rests on, and it is the one thing the
146
146
  correlation in Results 1 and 2 could not show. The kit does not shrink what it
147
147
  costs; it shrinks what the agent drags along behind it.
148
148
 
149
+ ## The A/B on this repository, 10 reps per arm
150
+
151
+ node eval/prepare.mjs --repo . --ref <sha> --name self
152
+ node eval/ab.mjs --target <scratch>/self --suite eval/tasks-self.json --reps 10
153
+
154
+ **The subject is this repository at a pinned commit**, so the run is
155
+ reproducible: `git clone && git checkout <sha>` and the same command. The
156
+ previous suite ran against a private repository, which meant a result nobody
157
+ else could check.
158
+
159
+ Six tasks, ten reps per arm, 120 agent runs: four lookups and two edits. The
160
+ lookups are graded on the `file:line` the agent cited, read out of the
161
+ snapshot; the edits are graded on the file after the run.
162
+
163
+ **Both arms ran on the same model**, `opencode/minimax-m3.1-flash-free`, and
164
+ the report prints it. The kit-off arm used to run with an empty config, so
165
+ opencode picked its own default: not the model the kit-on arm was on, and
166
+ not a model that answered - four kit-off runs hit the 600-second ceiling
167
+ with no output at all, and a timed-out run has zero tools and zero context,
168
+ so it entered the medians as a fast, empty, wrong answer.
169
+
170
+ | metric (median) | kit on | kit off |
171
+ | --- | --- | --- |
172
+ | **correct** | **54/60** | **55/60** |
173
+ | tool calls | 12.5 | 8 |
174
+ | read calls | 3 | 2 |
175
+ | narrow reads | 66.7% | 66.7% |
176
+ | grep/glob calls | 3 | 2 |
177
+ | steps | 9 | 6 |
178
+ | input tokens | 56,205 | 10,540 |
179
+ | final context | 39,928 | 12,718 |
180
+ | wall seconds | 43 | 21 |
181
+ | paths written outside the target | 0 | 0 |
182
+
183
+ | per task | kit on | kit off |
184
+ | --- | --- | --- |
185
+ | `read-control` | 10/10 | 10/10 |
186
+ | `read-multi-denylist` | 8/10 | 8/10 |
187
+ | `read-multi-gate` | 6/10 | 7/10 |
188
+ | `read-two-files` | 10/10 | 10/10 |
189
+ | `edit-rename` | 10/10 | 10/10 |
190
+ | `edit-multi` | 10/10 | 10/10 |
191
+
192
+ ### What this says
193
+
194
+ **On correctness, the kit is a tie.** 54 against 55, on sixty runs a side.
195
+ That is inside the noise of a single task flipping, and it is the opposite
196
+ of what the previous version of this file said. A second run of the same
197
+ 120 runs - same subject, same suite, same model, a different day of model
198
+ traffic - gave **55/60 against 54/60**, the other way. Two measurements
199
+ put the tie on either side of zero, which is what a tie looks like from
200
+ both ends.
201
+
202
+ **On everything else, the kit is a cost.** It spends **5.3x the input
203
+ tokens** and takes **twice as long**, and it makes **more** tool calls.
204
+ Those are the honest numbers.
205
+
206
+ **The previous conclusion was the grader's fault, and it is worth writing
207
+ down.** The earlier run said 52/60 against 59/59 - the kit looked clearly
208
+ worse - and the reason was that the grader could not read a citation
209
+ written as two markdown fields:
210
+
211
+ **File:** `src/profile.js`
212
+ **Line:** 105
213
+
214
+ The old pattern stopped at the newline, so the file and the number were
215
+ never paired, the citation was lost, and the answer was scored wrong. The
216
+ kit's own instructions tell the agent to cite a file and a line, and the
217
+ shape the kit produces is the shape the grader could not read. The kit-off
218
+ arm had no such instructions, answered in prose, and was graded by a
219
+ pattern that handled prose.
220
+
221
+ Re-scoring the EARLIER run with the fixed grader - the same 120 agent runs,
222
+ the same answers, only the grader changed - gives **34/40 against 30/40** on
223
+ the four lookup tasks, the kit ahead. A harness that measures the harness
224
+ is worse than one that measures nothing.
225
+
226
+ ### What this does not say
227
+
228
+ It does not say the kit is not worth it. It says the kit's value is **not in
229
+ short lookups**, and this suite only contains short lookups and two small
230
+ edits. The value it does have is in two places this suite cannot see:
231
+
232
+ - **Long sessions.** The pruner removes **65-80% of the context** across real
233
+ sessions, depending on which sessions it is measured over
234
+ (`npm run measure:prune`), and a task that fits in one read never gets
235
+ near the point where pruning matters. The long-task A/B below is the
236
+ measurement of that.
237
+ - **Process.** A `reviewer` that reviews and does not edit, a `verify` that
238
+ runs the checks and does not fix them, a `space-finish` that hands the
239
+ publish decision to a human. Those are behaviours, not accuracy, and a
240
+ grader that counts correct answers cannot see them. The process section
241
+ below measures what it can - and what it found is that the lane and the
242
+ default agent behave the same, and that a task saying "fix it" overrides
243
+ the lane.
244
+
245
+ The honest position is a trade, and it is now a trade with a number on it:
246
+ **the kit costs about five times the input tokens on a small task, gets the
247
+ answer right about as often, and buys process discipline and an 80% smaller
248
+ context on a long one.** Whether that is worth it depends on whether the
249
+ work is long. A repo of one-file lookups should not install this.
250
+
251
+ The narrow-read share is worth one more line. The kit's instructions say
252
+ *locate before you read*, and the share of reads that were bounded was
253
+ **the same on both sides, 66.7%** - the rule is followed, and on a
254
+ one-file question following it is waste either way. A fleet that routes
255
+ each job to the model built for it is the other half of what the kit
256
+ offers, and this suite uses one model on both sides, so it cannot see that.
257
+
149
258
  ## The A/B on TypeScript, against a PUBLIC subject
150
259
 
151
260
  node eval/prepare.mjs --clone https://github.com/typesense/typesense-js \
package/lanes.schema.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "$schema": "http://json-schema.org/draft-07/schema#",
3
- "$id": "https://github.com/pham-canh/space-harness/lanes.schema.json",
3
+ "$id": "https://github.com/pham-canh/opencode-harness-kit/lanes.schema.json",
4
4
  "title": "OpenCode harness lane table",
5
5
  "description": "The jobs a fleet can hand out. A lane says what it is for and nothing about models: a profile says which lanes it enables and which model serves each one. The lane table is the single list of lane names in the kit; `src/profile.js` reads it, `src/config.js` compiles it into the generated config, and `profiles/schema.json` must name the same lanes.",
6
6
  "type": "object",
package/package.json CHANGED
@@ -1,15 +1,15 @@
1
1
  {
2
2
  "name": "opencode-harness-kit",
3
- "version": "0.5.1",
3
+ "version": "0.5.4",
4
4
  "description": "Harness kit for OpenCode: capability profiles, a fleet of job lanes with one free model each, role-based reasoning effort, context pruning, and measured baselines.",
5
5
  "license": "MIT",
6
- "homepage": "https://github.com/pham-canh/space-harness#readme",
6
+ "homepage": "https://github.com/pham-canh/opencode-harness-kit#readme",
7
7
  "repository": {
8
8
  "type": "git",
9
- "url": "git+https://github.com/pham-canh/space-harness.git"
9
+ "url": "git+https://github.com/pham-canh/opencode-harness-kit.git"
10
10
  },
11
11
  "bugs": {
12
- "url": "https://github.com/pham-canh/space-harness/issues"
12
+ "url": "https://github.com/pham-canh/opencode-harness-kit/issues"
13
13
  },
14
14
  "type": "module",
15
15
  "main": "./dist/index.js",
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "$schema": "http://json-schema.org/draft-07/schema#",
3
- "$id": "https://github.com/pham-canh/space-harness/profiles/schema.json",
3
+ "$id": "https://github.com/pham-canh/opencode-harness-kit/profiles/schema.json",
4
4
  "title": "OpenCode harness profile",
5
5
  "description": "Everything that changes when you change model. Plugins read this file and contain no model names of their own.",
6
6
  "type": "object",