opencode-harness-kit 0.5.1 → 0.5.4
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +127 -17
- package/CONTRIBUTING.md +4 -4
- package/README.md +75 -10
- package/SECURITY.md +2 -2
- package/bin/oc-harness.mjs +20 -7
- package/eval/RESULTS.md +109 -0
- package/lanes.schema.json +1 -1
- package/package.json +4 -4
- package/profiles/schema.json +1 -1
package/CHANGELOG.md
CHANGED
|
@@ -12,6 +12,109 @@ is not happening.
|
|
|
12
12
|
|
|
13
13
|
---
|
|
14
14
|
|
|
15
|
+
## [0.5.4] — 2026-10-21
|
|
16
|
+
|
|
17
|
+
Published. The tag is what cut it: `git tag v0.5.4 && git push origin main
|
|
18
|
+
v0.5.4`, and the publish job watches for exactly that.
|
|
19
|
+
|
|
20
|
+
A patch, and the patch is the publish path. 0.5.3 was the first release to
|
|
21
|
+
reach the registry since the rename, and it reached it through trusted
|
|
22
|
+
publishing with a signed provenance statement - so the path works, and this
|
|
23
|
+
release is the one that says so where a reader will look for it.
|
|
24
|
+
|
|
25
|
+
### Changed
|
|
26
|
+
|
|
27
|
+
- **The npm trusted publisher is configured for
|
|
28
|
+
`pham-canh/opencode-harness-kit`**: `ci.yml`, no environment, publish and
|
|
29
|
+
stage. It is the one thing that was missing when 0.5.2 was refused, and it
|
|
30
|
+
is set now. The re-point is an account change on npm, so it needed the
|
|
31
|
+
account's second factor - and a rename is not a thing npm does for you.
|
|
32
|
+
|
|
33
|
+
- **The publish token secret is gone.** The job can publish with a token
|
|
34
|
+
when there is one, and the secret was set while 0.5.3 was cut, because a
|
|
35
|
+
release that cannot publish is not a release. It is removed now that the
|
|
36
|
+
OIDC path is proven: a long-lived token in the repository is a secret to
|
|
37
|
+
rotate for a path that is not being used, and npm is restricting the
|
|
38
|
+
tokens that bypass two-factor authentication for direct publishing in
|
|
39
|
+
January 2027, so the token path was on its way out anyway.
|
|
40
|
+
|
|
41
|
+
## [0.5.3] — 2026-10-21
|
|
42
|
+
|
|
43
|
+
Published. The tag is what cut it: `git tag v0.5.3 && git push origin main
|
|
44
|
+
v0.5.3`, and the publish job watches for exactly that.
|
|
45
|
+
|
|
46
|
+
This is the release that carries the rename, because 0.5.2 - the release the
|
|
47
|
+
rename was written for - never reached the registry.
|
|
48
|
+
|
|
49
|
+
### Changed
|
|
50
|
+
|
|
51
|
+
- **The repository is now `pham-canh/opencode-harness-kit`**, renamed from
|
|
52
|
+
`pham-canh/space-harness`. The npm package has been
|
|
53
|
+
`opencode-harness-kit` since 0.4.0, and the README's title with it; the
|
|
54
|
+
repository was the last one still carrying the old name, and a reader who
|
|
55
|
+
installed the package and followed the link landed somewhere that did not
|
|
56
|
+
look like the thing they had installed. GitHub redirects the old URL, so
|
|
57
|
+
the links that already exist keep working.
|
|
58
|
+
|
|
59
|
+
The npm page showed the old name until this release, and the reason is
|
|
60
|
+
worth writing down, because nothing about the rename explains it: the page
|
|
61
|
+
reads the `repository` and `homepage` out of the manifest of the LATEST
|
|
62
|
+
PUBLISHED version, and the latest published version was 0.5.1, whose
|
|
63
|
+
manifest was written before the rename. The manifest names the new
|
|
64
|
+
repository now, and this is the release that carries it.
|
|
65
|
+
|
|
66
|
+
### Fixed
|
|
67
|
+
|
|
68
|
+
- **The publish job now checks the OIDC claim against `package.json` before
|
|
69
|
+
it publishes.** npm matches the trusted publisher's repository against the
|
|
70
|
+
claim, and the claim names the repository the workflow ran in - so a
|
|
71
|
+
renamed repository stops publishing, with an exchange error that does not
|
|
72
|
+
name the field that was wrong. The gate says which side is which and
|
|
73
|
+
prints the command that fixes it.
|
|
74
|
+
|
|
75
|
+
- **The installer no longer says "installed" when it replaced nothing.**
|
|
76
|
+
`existsSync` follows a symlink, so a link whose target no longer exists
|
|
77
|
+
reads as "not there" - and the branch that replaces what the installer put
|
|
78
|
+
there was never reached. `symlinkSync` then threw EEXIST over the link
|
|
79
|
+
that was already there, the warning was printed where nobody reads it, and
|
|
80
|
+
the install carried on. After the rename that is exactly the state a kit
|
|
81
|
+
installed from the old path was in: the skills, the prompts and GLOBAL.md
|
|
82
|
+
kept pointing at a directory that had moved, and `verify` then found
|
|
83
|
+
seventeen broken paths. `lstat` does not follow the link, and it is what
|
|
84
|
+
decides whether there is something to replace.
|
|
85
|
+
|
|
86
|
+
### Removed
|
|
87
|
+
|
|
88
|
+
- **The coverage report is no longer committed.** 131 files of it, four
|
|
89
|
+
megabytes, changing on every run, and nobody reads a diff of it. It is
|
|
90
|
+
ignored now and produced on demand.
|
|
91
|
+
|
|
92
|
+
- **The two shell scripts are gone from the repository**: the demo setup
|
|
93
|
+
helper, which pointed at copies of two tools that live one directory up
|
|
94
|
+
with their own history, and the take script that filmed the published
|
|
95
|
+
demo, which is a shell file with one machine's paths in it and goes stale
|
|
96
|
+
the moment the thing it films changes. The recording guide is on the
|
|
97
|
+
machine that records.
|
|
98
|
+
|
|
99
|
+
## [0.5.2] — 2026-10-21
|
|
100
|
+
|
|
101
|
+
**Not published.** The tag was cut and pushed, the publish job ran, and npm
|
|
102
|
+
refused the publish - so there is no 0.5.2 on the registry. The changes
|
|
103
|
+
written for it shipped in 0.5.3.
|
|
104
|
+
|
|
105
|
+
What npm said was `OIDC token exchange error - package not found`, which
|
|
106
|
+
reads like a missing package. The package was there. The trusted publisher
|
|
107
|
+
was not: npm reports no trusted publisher configured for this package, and
|
|
108
|
+
the repository had been renamed, so a publisher that named the old
|
|
109
|
+
repository would not have matched the claim the runner presents either. The
|
|
110
|
+
package's name is in the error, and the field that was wrong is not.
|
|
111
|
+
|
|
112
|
+
The entry this release was going to carry said that "the npm trusted
|
|
113
|
+
publisher was re-pointed at the new name". It had not been. The re-point is
|
|
114
|
+
an account change on npm and it needs the account's second factor, and
|
|
115
|
+
nothing had done it. A changelog that says a thing has been done that has
|
|
116
|
+
not been done is worse than one that says nothing.
|
|
117
|
+
|
|
15
118
|
## [0.5.1] — 2026-10-21
|
|
16
119
|
|
|
17
120
|
Published. The tag is what cut it: `git tag v0.5.1 && git push origin main
|
|
@@ -1130,20 +1233,27 @@ silent in the worst way:
|
|
|
1130
1233
|
|
|
1131
1234
|
---
|
|
1132
1235
|
|
|
1133
|
-
[Unreleased]: https://github.com/pham-canh/
|
|
1134
|
-
[0.
|
|
1135
|
-
[0.
|
|
1136
|
-
[0.
|
|
1137
|
-
[0.
|
|
1138
|
-
[0.
|
|
1139
|
-
[0.
|
|
1140
|
-
[0.
|
|
1141
|
-
[0.
|
|
1142
|
-
[0.3
|
|
1143
|
-
[0.
|
|
1144
|
-
[0.
|
|
1145
|
-
[0.
|
|
1146
|
-
[0.3.
|
|
1147
|
-
[0.
|
|
1148
|
-
[0.
|
|
1149
|
-
[0.
|
|
1236
|
+
[Unreleased]: https://github.com/pham-canh/opencode-harness-kit/compare/v0.5.4...HEAD
|
|
1237
|
+
[0.5.4]: https://github.com/pham-canh/opencode-harness-kit/compare/v0.5.3...v0.5.4
|
|
1238
|
+
[0.5.3]: https://github.com/pham-canh/opencode-harness-kit/compare/v0.5.2...v0.5.3
|
|
1239
|
+
[0.5.2]: https://github.com/pham-canh/opencode-harness-kit/compare/v0.5.1...v0.5.2
|
|
1240
|
+
[0.5.1]: https://github.com/pham-canh/opencode-harness-kit/compare/v0.5.0...v0.5.1
|
|
1241
|
+
[0.5.0]: https://github.com/pham-canh/opencode-harness-kit/compare/v0.4.6...v0.5.0
|
|
1242
|
+
[0.4.6]: https://github.com/pham-canh/opencode-harness-kit/compare/v0.4.5...v0.4.6
|
|
1243
|
+
[0.4.5]: https://github.com/pham-canh/opencode-harness-kit/compare/v0.4.4...v0.4.5
|
|
1244
|
+
[0.4.4]: https://github.com/pham-canh/opencode-harness-kit/compare/v0.4.3...v0.4.4
|
|
1245
|
+
[0.4.3]: https://github.com/pham-canh/opencode-harness-kit/compare/v0.4.2...v0.4.3
|
|
1246
|
+
[0.4.2]: https://github.com/pham-canh/opencode-harness-kit/compare/v0.4.1...v0.4.2
|
|
1247
|
+
[0.4.1]: https://github.com/pham-canh/opencode-harness-kit/compare/v0.4.0...v0.4.1
|
|
1248
|
+
[0.4.0]: https://github.com/pham-canh/opencode-harness-kit/compare/v0.3.9...v0.4.0
|
|
1249
|
+
[0.3.9]: https://github.com/pham-canh/opencode-harness-kit/compare/v0.3.8...v0.3.9
|
|
1250
|
+
[0.3.8]: https://github.com/pham-canh/opencode-harness-kit/compare/v0.3.7...v0.3.8
|
|
1251
|
+
[0.3.7]: https://github.com/pham-canh/opencode-harness-kit/compare/v0.3.6...v0.3.7
|
|
1252
|
+
[0.3.6]: https://github.com/pham-canh/opencode-harness-kit/compare/v0.3.5...v0.3.6
|
|
1253
|
+
[0.3.5]: https://github.com/pham-canh/opencode-harness-kit/compare/v0.3.4...v0.3.5
|
|
1254
|
+
[0.3.4]: https://github.com/pham-canh/opencode-harness-kit/compare/v0.3.1...v0.3.4
|
|
1255
|
+
[0.3.1]: https://github.com/pham-canh/opencode-harness-kit/compare/v0.3.0...v0.3.1
|
|
1256
|
+
[0.3.0]: https://github.com/pham-canh/opencode-harness-kit/compare/v0.2.1...v0.3.0
|
|
1257
|
+
[0.2.1]: https://github.com/pham-canh/opencode-harness-kit/compare/v0.2.0...v0.2.1
|
|
1258
|
+
[0.2.0]: https://github.com/pham-canh/opencode-harness-kit/compare/v0.1.0...v0.2.0
|
|
1259
|
+
[0.1.0]: https://github.com/pham-canh/opencode-harness-kit/releases/tag/v0.1.0
|
package/CONTRIBUTING.md
CHANGED
|
@@ -139,7 +139,7 @@ repository secret to a granular token scoped to `opencode-harness-kit` with
|
|
|
139
139
|
if it is absent the job uses OIDC.
|
|
140
140
|
|
|
141
141
|
```bash
|
|
142
|
-
gh secret set NPM_TOKEN --repo pham-canh/
|
|
142
|
+
gh secret set NPM_TOKEN --repo pham-canh/opencode-harness-kit
|
|
143
143
|
```
|
|
144
144
|
|
|
145
145
|
> **`gh secret set` takes the secret NAME as its first argument**, and the
|
|
@@ -213,7 +213,7 @@ To set it without the script, the command is:
|
|
|
213
213
|
```bash
|
|
214
214
|
npm trust github opencode-harness-kit \
|
|
215
215
|
--file ci.yml \
|
|
216
|
-
--repo pham-canh/
|
|
216
|
+
--repo pham-canh/opencode-harness-kit \
|
|
217
217
|
--allow-publish --allow-stage-publish -y
|
|
218
218
|
```
|
|
219
219
|
|
|
@@ -222,7 +222,7 @@ The configuration it writes is:
|
|
|
222
222
|
| field | value |
|
|
223
223
|
|---|---|
|
|
224
224
|
| Organization or user | `pham-canh` |
|
|
225
|
-
| Repository | `
|
|
225
|
+
| Repository | `opencode-harness-kit` |
|
|
226
226
|
| Workflow filename | `ci.yml` (the filename only, not a path) |
|
|
227
227
|
| Environment name | **empty** |
|
|
228
228
|
| Allowed actions | `npm publish` and `npm stage publish` |
|
|
@@ -236,7 +236,7 @@ The workflow is the one in `.github/workflows/ci.yml`, and the claims it
|
|
|
236
236
|
presents are printed by the publish job before it tries:
|
|
237
237
|
|
|
238
238
|
```
|
|
239
|
-
repository pham-canh/
|
|
239
|
+
repository pham-canh/opencode-harness-kit
|
|
240
240
|
workflow file ci.yml
|
|
241
241
|
environment (absent) - the trusted publisher must name NO environment
|
|
242
242
|
owner pham-canh
|
package/README.md
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
# opencode-harness-kit
|
|
2
2
|
|
|
3
|
-
[](https://github.com/pham-canh/opencode-harness-kit/actions/workflows/ci.yml)
|
|
4
4
|
[](https://www.npmjs.com/package/opencode-harness-kit)
|
|
5
5
|
|
|
6
6
|
<p align="center">
|
|
@@ -8,9 +8,9 @@
|
|
|
8
8
|
</p>
|
|
9
9
|
|
|
10
10
|
*Four minutes forty-seven seconds, recorded live from a real terminal and
|
|
11
|
-
narrated
|
|
12
|
-
|
|
13
|
-
|
|
11
|
+
narrated — every command on screen is typed and executed while the camera
|
|
12
|
+
runs. The voiceover is in [`video/`](video/), along with what it took to
|
|
13
|
+
record it and the two mistakes that cost the most time.*
|
|
14
14
|
|
|
15
15
|
*It was recorded against an earlier version of the kit: the numbers it names
|
|
16
16
|
are the ones from then, and the ones below are the ones now.*
|
|
@@ -31,7 +31,7 @@ your side — `npx` only downloads the published kit. Prefer git? The same three
|
|
|
31
31
|
steps from a checkout (see [Install](#install)):
|
|
32
32
|
|
|
33
33
|
```bash
|
|
34
|
-
git clone https://github.com/pham-canh/
|
|
34
|
+
git clone https://github.com/pham-canh/opencode-harness-kit.git
|
|
35
35
|
./bin/oc-harness.mjs init
|
|
36
36
|
```
|
|
37
37
|
|
|
@@ -67,6 +67,33 @@ src/ ← no model name anywhere in here
|
|
|
67
67
|
Same plugin source, twelve shipped profiles, twelve different behaviours. The
|
|
68
68
|
test suite asserts that rather than taking it on faith.
|
|
69
69
|
|
|
70
|
+
### How a request travels
|
|
71
|
+
|
|
72
|
+
```mermaid
|
|
73
|
+
flowchart LR
|
|
74
|
+
P[profile JSON<br/>model, effort, fallback, retention] --> C[oc-harness use<br/>compiles the profile]
|
|
75
|
+
C --> CFG[~/.config/opencode/opencode.jsonc<br/>agent.<lane>.model]
|
|
76
|
+
CFG --> PL[the plugin: guard, prune, router, telemetry]
|
|
77
|
+
PL --> LANES[10 lanes<br/>each on the model built for it]
|
|
78
|
+
LANES --> PR[prune: old tool output<br/>becomes a stub]
|
|
79
|
+
LANES --> GD[guard: denied commands<br/>and writes outside the project]
|
|
80
|
+
LANES --> RT[router: a spent route<br/>moves to the next rung]
|
|
81
|
+
LANES --> TL[telemetry: what each lane<br/>actually cost]
|
|
82
|
+
```
|
|
83
|
+
|
|
84
|
+
Four things happen to a request, and they are four separate files:
|
|
85
|
+
|
|
86
|
+
| | |
|
|
87
|
+
|---|---|
|
|
88
|
+
| `src/guard.ts` | refuses the commands the kit never allows, and any write outside the project |
|
|
89
|
+
| `src/prune.ts` | replaces old tool output with a one-line stub, so the window holds the work and not the noise |
|
|
90
|
+
| `src/router.ts` | when a route is spent, moves the lane to the next rung of its own chain |
|
|
91
|
+
| `src/telemetry.ts` | records what each lane actually cost, so `oc-harness lanes` can say which ones earn their place |
|
|
92
|
+
|
|
93
|
+
The model for a lane is in the **config**, not in a prompt and not in a file
|
|
94
|
+
the agent reads. That is the whole design: the answers are data, and a data
|
|
95
|
+
edit is reviewable.
|
|
96
|
+
|
|
70
97
|
## A fleet, not a model
|
|
71
98
|
|
|
72
99
|
The default profile is a **fleet**: ten jobs, ten free models, one session.
|
|
@@ -690,8 +717,8 @@ Without registry access, the checkout path works identically:
|
|
|
690
717
|
|
|
691
718
|
```bash
|
|
692
719
|
pkg install nodejs git
|
|
693
|
-
git clone https://github.com/pham-canh/
|
|
694
|
-
./
|
|
720
|
+
git clone https://github.com/pham-canh/opencode-harness-kit.git
|
|
721
|
+
./opencode-harness-kit/bin/oc-harness.mjs init
|
|
695
722
|
```
|
|
696
723
|
|
|
697
724
|
The kit writes to `$HOME/.config/opencode` like everywhere else, so there is
|
|
@@ -863,6 +890,38 @@ session, so switching happens at the next session, not mid-flight. What a
|
|
|
863
890
|
plugin *can* do is move a lane to the next rung of its chain before the
|
|
864
891
|
session starts.
|
|
865
892
|
|
|
893
|
+
## What is next, and what is not measured yet
|
|
894
|
+
|
|
895
|
+
The claims this project makes are measured, and where a measurement has not
|
|
896
|
+
happened the claim is written down as a gap rather than as a fact. These are
|
|
897
|
+
the open ones.
|
|
898
|
+
|
|
899
|
+
**The pruner's rule was measured on one task.** Keeping the newest copy of
|
|
900
|
+
each distinct tool output instead of a window on recent calls cut the input
|
|
901
|
+
tokens by 69% and the wall time by 60% on the long-task A/B. One task is one
|
|
902
|
+
workload. It needs the same treatment on a repository that is not this one,
|
|
903
|
+
and on a session that runs for hours rather than minutes.
|
|
904
|
+
|
|
905
|
+
**The reviewer's "you never fix" is an instruction, not a boundary.** The
|
|
906
|
+
measurement of it came out even: the kit's reviewer lane and a default agent
|
|
907
|
+
behaved the same on a review task, and a task that says "fix it" overrode
|
|
908
|
+
the lane. The kit's own loop keeps the line by never asking a reviewer to
|
|
909
|
+
fix anything. A guard that cannot be overridden by a task is worth having,
|
|
910
|
+
and that is not built yet.
|
|
911
|
+
|
|
912
|
+
**The nightly A/B needs a provider key on the repository.** The workflow
|
|
913
|
+
exists, it runs, and it is skipped until the key is set — the variable
|
|
914
|
+
`HAS_OPENCODE_KEY` says the secret is there. The numbers in
|
|
915
|
+
[eval/RESULTS.md](eval/RESULTS.md) are the ones taken by hand.
|
|
916
|
+
|
|
917
|
+
**The TypeScript pilot is one public repository.** It is now a real one
|
|
918
|
+
(`typesense-js`, pinned by commit) rather than a private tree, which is what
|
|
919
|
+
made it checkable. Two languages are not a survey.
|
|
920
|
+
|
|
921
|
+
**The lanes are ten because ten models were available.** That is a
|
|
922
|
+
hypothesis about the tier, not a number. `oc-harness lanes` counts the
|
|
923
|
+
requests and says which lanes were never used; cutting them is a data edit.
|
|
924
|
+
|
|
866
925
|
## Development
|
|
867
926
|
|
|
868
927
|
```bash
|
|
@@ -907,9 +966,15 @@ and the measurements behind each choice.
|
|
|
907
966
|
|
|
908
967
|
[`CHANGELOG.md`](CHANGELOG.md) records what each release changed and, for the
|
|
909
968
|
releases before this one, what was broken in it — the silent failures that a
|
|
910
|
-
config could look correct through. [`MIGRATION.md`](MIGRATION.md) covers
|
|
911
|
-
0.2.x → 0.3.0
|
|
912
|
-
telemetry file now means something
|
|
969
|
+
config could look correct through. [`MIGRATION.md`](MIGRATION.md) covers the
|
|
970
|
+
two upgrades that ask something of you: 0.2.x → 0.3.0 (the profile ids
|
|
971
|
+
changed, the config layout changed, the telemetry file now means something
|
|
972
|
+
different) and 0.3.x → 0.4.x (the guard started refusing in code what it used
|
|
973
|
+
to only suggest, and the pre-commit hook started running your project's
|
|
974
|
+
tests instead of the kit's).
|
|
975
|
+
|
|
976
|
+
[`docs/EXIT-CODES.md`](docs/EXIT-CODES.md) lists what every command returns
|
|
977
|
+
when it fails, so a script can tell "the kit says no" from "the kit broke".
|
|
913
978
|
|
|
914
979
|
## Contributing
|
|
915
980
|
|
package/SECURITY.md
CHANGED
|
@@ -12,7 +12,7 @@
|
|
|
12
12
|
## Reporting a vulnerability
|
|
13
13
|
|
|
14
14
|
Please report privately through GitHub's
|
|
15
|
-
[private vulnerability reporting](https://github.com/pham-canh/
|
|
15
|
+
[private vulnerability reporting](https://github.com/pham-canh/opencode-harness-kit/security/advisories/new)
|
|
16
16
|
rather than opening a public issue.
|
|
17
17
|
|
|
18
18
|
Include the affected version, a reproduction, and the expected behaviour.
|
|
@@ -21,7 +21,7 @@ plan within 7 days.
|
|
|
21
21
|
|
|
22
22
|
## Scope
|
|
23
23
|
|
|
24
|
-
`
|
|
24
|
+
`opencode-harness-kit` runs agent subprocesses and reads harness config. Reports
|
|
25
25
|
that are in scope include path traversal when resolving a capability
|
|
26
26
|
profile, command injection through harness config values, and any way a
|
|
27
27
|
harness writes outside its configured working directory.
|
package/bin/oc-harness.mjs
CHANGED
|
@@ -706,19 +706,32 @@ function linkOne(src, dest) {
|
|
|
706
706
|
}
|
|
707
707
|
if (ours) return
|
|
708
708
|
|
|
709
|
-
if (
|
|
709
|
+
if (isLink) {
|
|
710
|
+
// A symlink is a symlink whether or not its target exists, and a
|
|
711
|
+
// DANGLING one is still this installer's to replace.
|
|
712
|
+
//
|
|
713
|
+
// `existsSync` follows the link, so it says false for a link whose
|
|
714
|
+
// source has moved - the repository was renamed, or the kit was
|
|
715
|
+
// reinstalled somewhere else. The old code asked `existsSync` whether
|
|
716
|
+
// the destination was taken, got "no", and then tried to create a link
|
|
717
|
+
// over an existing one: EEXIST, printed as a warning nobody reads, and
|
|
718
|
+
// the install reported success. Every skill, prompt and rule kept
|
|
719
|
+
// pointing at a directory that is no longer there, and `verify` found
|
|
720
|
+
// seventeen broken paths afterwards.
|
|
721
|
+
//
|
|
722
|
+
// The test that decides whether to replace something is `lstat`, which
|
|
723
|
+
// does not follow the link.
|
|
724
|
+
rmSync(dest, { force: true })
|
|
725
|
+
} else if (existsSync(dest)) {
|
|
710
726
|
// Replace only what this installer put there. The old code treated "not
|
|
711
727
|
// already our link" as "safe to delete" and called `rmSync` with
|
|
712
728
|
// `recursive: false`, which succeeds on a plain file: a hand-written
|
|
713
729
|
// agent was deleted and replaced by a symlink without a word. A real
|
|
714
730
|
// directory is left alone too -- unlinking it is not this installer's
|
|
715
731
|
// call to make.
|
|
716
|
-
|
|
717
|
-
|
|
718
|
-
|
|
719
|
-
return
|
|
720
|
-
}
|
|
721
|
-
rmSync(dest, { force: true })
|
|
732
|
+
const kind = lstatSync(dest).isDirectory() ? "directory" : "file"
|
|
733
|
+
console.log(` ${yellow("!")} ${stamp(`left alone: ${dest} is a real ${kind} and was not ours to replace`)}`)
|
|
734
|
+
return
|
|
722
735
|
}
|
|
723
736
|
|
|
724
737
|
try {
|
package/eval/RESULTS.md
CHANGED
|
@@ -146,6 +146,115 @@ That is the mechanism the whole kit rests on, and it is the one thing the
|
|
|
146
146
|
correlation in Results 1 and 2 could not show. The kit does not shrink what it
|
|
147
147
|
costs; it shrinks what the agent drags along behind it.
|
|
148
148
|
|
|
149
|
+
## The A/B on this repository, 10 reps per arm
|
|
150
|
+
|
|
151
|
+
node eval/prepare.mjs --repo . --ref <sha> --name self
|
|
152
|
+
node eval/ab.mjs --target <scratch>/self --suite eval/tasks-self.json --reps 10
|
|
153
|
+
|
|
154
|
+
**The subject is this repository at a pinned commit**, so the run is
|
|
155
|
+
reproducible: `git clone && git checkout <sha>` and the same command. The
|
|
156
|
+
previous suite ran against a private repository, which meant a result nobody
|
|
157
|
+
else could check.
|
|
158
|
+
|
|
159
|
+
Six tasks, ten reps per arm, 120 agent runs: four lookups and two edits. The
|
|
160
|
+
lookups are graded on the `file:line` the agent cited, read out of the
|
|
161
|
+
snapshot; the edits are graded on the file after the run.
|
|
162
|
+
|
|
163
|
+
**Both arms ran on the same model**, `opencode/minimax-m3.1-flash-free`, and
|
|
164
|
+
the report prints it. The kit-off arm used to run with an empty config, so
|
|
165
|
+
opencode picked its own default: not the model the kit-on arm was on, and
|
|
166
|
+
not a model that answered - four kit-off runs hit the 600-second ceiling
|
|
167
|
+
with no output at all, and a timed-out run has zero tools and zero context,
|
|
168
|
+
so it entered the medians as a fast, empty, wrong answer.
|
|
169
|
+
|
|
170
|
+
| metric (median) | kit on | kit off |
|
|
171
|
+
| --- | --- | --- |
|
|
172
|
+
| **correct** | **54/60** | **55/60** |
|
|
173
|
+
| tool calls | 12.5 | 8 |
|
|
174
|
+
| read calls | 3 | 2 |
|
|
175
|
+
| narrow reads | 66.7% | 66.7% |
|
|
176
|
+
| grep/glob calls | 3 | 2 |
|
|
177
|
+
| steps | 9 | 6 |
|
|
178
|
+
| input tokens | 56,205 | 10,540 |
|
|
179
|
+
| final context | 39,928 | 12,718 |
|
|
180
|
+
| wall seconds | 43 | 21 |
|
|
181
|
+
| paths written outside the target | 0 | 0 |
|
|
182
|
+
|
|
183
|
+
| per task | kit on | kit off |
|
|
184
|
+
| --- | --- | --- |
|
|
185
|
+
| `read-control` | 10/10 | 10/10 |
|
|
186
|
+
| `read-multi-denylist` | 8/10 | 8/10 |
|
|
187
|
+
| `read-multi-gate` | 6/10 | 7/10 |
|
|
188
|
+
| `read-two-files` | 10/10 | 10/10 |
|
|
189
|
+
| `edit-rename` | 10/10 | 10/10 |
|
|
190
|
+
| `edit-multi` | 10/10 | 10/10 |
|
|
191
|
+
|
|
192
|
+
### What this says
|
|
193
|
+
|
|
194
|
+
**On correctness, the kit is a tie.** 54 against 55, on sixty runs a side.
|
|
195
|
+
That is inside the noise of a single task flipping, and it is the opposite
|
|
196
|
+
of what the previous version of this file said. A second run of the same
|
|
197
|
+
120 runs - same subject, same suite, same model, a different day of model
|
|
198
|
+
traffic - gave **55/60 against 54/60**, the other way. Two measurements
|
|
199
|
+
put the tie on either side of zero, which is what a tie looks like from
|
|
200
|
+
both ends.
|
|
201
|
+
|
|
202
|
+
**On everything else, the kit is a cost.** It spends **5.3x the input
|
|
203
|
+
tokens** and takes **twice as long**, and it makes **more** tool calls.
|
|
204
|
+
Those are the honest numbers.
|
|
205
|
+
|
|
206
|
+
**The previous conclusion was the grader's fault, and it is worth writing
|
|
207
|
+
down.** The earlier run said 52/60 against 59/59 - the kit looked clearly
|
|
208
|
+
worse - and the reason was that the grader could not read a citation
|
|
209
|
+
written as two markdown fields:
|
|
210
|
+
|
|
211
|
+
**File:** `src/profile.js`
|
|
212
|
+
**Line:** 105
|
|
213
|
+
|
|
214
|
+
The old pattern stopped at the newline, so the file and the number were
|
|
215
|
+
never paired, the citation was lost, and the answer was scored wrong. The
|
|
216
|
+
kit's own instructions tell the agent to cite a file and a line, and the
|
|
217
|
+
shape the kit produces is the shape the grader could not read. The kit-off
|
|
218
|
+
arm had no such instructions, answered in prose, and was graded by a
|
|
219
|
+
pattern that handled prose.
|
|
220
|
+
|
|
221
|
+
Re-scoring the EARLIER run with the fixed grader - the same 120 agent runs,
|
|
222
|
+
the same answers, only the grader changed - gives **34/40 against 30/40** on
|
|
223
|
+
the four lookup tasks, the kit ahead. A harness that measures the harness
|
|
224
|
+
is worse than one that measures nothing.
|
|
225
|
+
|
|
226
|
+
### What this does not say
|
|
227
|
+
|
|
228
|
+
It does not say the kit is not worth it. It says the kit's value is **not in
|
|
229
|
+
short lookups**, and this suite only contains short lookups and two small
|
|
230
|
+
edits. The value it does have is in two places this suite cannot see:
|
|
231
|
+
|
|
232
|
+
- **Long sessions.** The pruner removes **65-80% of the context** across real
|
|
233
|
+
sessions, depending on which sessions it is measured over
|
|
234
|
+
(`npm run measure:prune`), and a task that fits in one read never gets
|
|
235
|
+
near the point where pruning matters. The long-task A/B below is the
|
|
236
|
+
measurement of that.
|
|
237
|
+
- **Process.** A `reviewer` that reviews and does not edit, a `verify` that
|
|
238
|
+
runs the checks and does not fix them, a `space-finish` that hands the
|
|
239
|
+
publish decision to a human. Those are behaviours, not accuracy, and a
|
|
240
|
+
grader that counts correct answers cannot see them. The process section
|
|
241
|
+
below measures what it can - and what it found is that the lane and the
|
|
242
|
+
default agent behave the same, and that a task saying "fix it" overrides
|
|
243
|
+
the lane.
|
|
244
|
+
|
|
245
|
+
The honest position is a trade, and it is now a trade with a number on it:
|
|
246
|
+
**the kit costs about five times the input tokens on a small task, gets the
|
|
247
|
+
answer right about as often, and buys process discipline and an 80% smaller
|
|
248
|
+
context on a long one.** Whether that is worth it depends on whether the
|
|
249
|
+
work is long. A repo of one-file lookups should not install this.
|
|
250
|
+
|
|
251
|
+
The narrow-read share is worth one more line. The kit's instructions say
|
|
252
|
+
*locate before you read*, and the share of reads that were bounded was
|
|
253
|
+
**the same on both sides, 66.7%** - the rule is followed, and on a
|
|
254
|
+
one-file question following it is waste either way. A fleet that routes
|
|
255
|
+
each job to the model built for it is the other half of what the kit
|
|
256
|
+
offers, and this suite uses one model on both sides, so it cannot see that.
|
|
257
|
+
|
|
149
258
|
## The A/B on TypeScript, against a PUBLIC subject
|
|
150
259
|
|
|
151
260
|
node eval/prepare.mjs --clone https://github.com/typesense/typesense-js \
|
package/lanes.schema.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"$schema": "http://json-schema.org/draft-07/schema#",
|
|
3
|
-
"$id": "https://github.com/pham-canh/
|
|
3
|
+
"$id": "https://github.com/pham-canh/opencode-harness-kit/lanes.schema.json",
|
|
4
4
|
"title": "OpenCode harness lane table",
|
|
5
5
|
"description": "The jobs a fleet can hand out. A lane says what it is for and nothing about models: a profile says which lanes it enables and which model serves each one. The lane table is the single list of lane names in the kit; `src/profile.js` reads it, `src/config.js` compiles it into the generated config, and `profiles/schema.json` must name the same lanes.",
|
|
6
6
|
"type": "object",
|
package/package.json
CHANGED
|
@@ -1,15 +1,15 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "opencode-harness-kit",
|
|
3
|
-
"version": "0.5.
|
|
3
|
+
"version": "0.5.4",
|
|
4
4
|
"description": "Harness kit for OpenCode: capability profiles, a fleet of job lanes with one free model each, role-based reasoning effort, context pruning, and measured baselines.",
|
|
5
5
|
"license": "MIT",
|
|
6
|
-
"homepage": "https://github.com/pham-canh/
|
|
6
|
+
"homepage": "https://github.com/pham-canh/opencode-harness-kit#readme",
|
|
7
7
|
"repository": {
|
|
8
8
|
"type": "git",
|
|
9
|
-
"url": "git+https://github.com/pham-canh/
|
|
9
|
+
"url": "git+https://github.com/pham-canh/opencode-harness-kit.git"
|
|
10
10
|
},
|
|
11
11
|
"bugs": {
|
|
12
|
-
"url": "https://github.com/pham-canh/
|
|
12
|
+
"url": "https://github.com/pham-canh/opencode-harness-kit/issues"
|
|
13
13
|
},
|
|
14
14
|
"type": "module",
|
|
15
15
|
"main": "./dist/index.js",
|
package/profiles/schema.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"$schema": "http://json-schema.org/draft-07/schema#",
|
|
3
|
-
"$id": "https://github.com/pham-canh/
|
|
3
|
+
"$id": "https://github.com/pham-canh/opencode-harness-kit/profiles/schema.json",
|
|
4
4
|
"title": "OpenCode harness profile",
|
|
5
5
|
"description": "Everything that changes when you change model. Plugins read this file and contain no model names of their own.",
|
|
6
6
|
"type": "object",
|