opencode-harness-kit 0.5.0 → 0.5.3
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +150 -17
- package/CONTRIBUTING.md +11 -4
- package/README.md +76 -11
- package/SECURITY.md +2 -2
- package/bin/oc-harness.mjs +20 -7
- package/eval/RESULTS.md +116 -7
- package/eval/verify.mjs +28 -4
- package/lanes.schema.json +1 -1
- package/package.json +4 -4
- package/profiles/schema.json +1 -1
package/CHANGELOG.md
CHANGED
|
@@ -12,6 +12,133 @@ is not happening.
|
|
|
12
12
|
|
|
13
13
|
---
|
|
14
14
|
|
|
15
|
+
## [0.5.2] — 2026-10-21
|
|
16
|
+
|
|
17
|
+
**Not published.** The tag was cut and pushed, the publish job ran, and npm
|
|
18
|
+
refused the publish - so there is no 0.5.2 on the registry. The changes
|
|
19
|
+
written for it shipped in 0.5.3.
|
|
20
|
+
|
|
21
|
+
What npm said was `OIDC token exchange error - package not found`, which
|
|
22
|
+
reads like a missing package. The package was there; the trusted publisher
|
|
23
|
+
was not, because npm looks a publisher up by (package, repository) and the
|
|
24
|
+
repository had just been renamed. The stored entry still named
|
|
25
|
+
`pham-canh/space-harness`; the claim the runner presents names
|
|
26
|
+
`pham-canh/opencode-harness-kit`. The package's name is in the error, and
|
|
27
|
+
the field that was wrong is not.
|
|
28
|
+
|
|
29
|
+
The entry this release was going to carry said that "the npm trusted
|
|
30
|
+
publisher was re-pointed at the new name". It had not been. The re-point is
|
|
31
|
+
an account change on npm and it needs the account's second factor, and a
|
|
32
|
+
changelog that says a thing has been done that has not been done is worse
|
|
33
|
+
than one that says nothing.
|
|
34
|
+
|
|
35
|
+
## [0.5.3] — 2026-10-21
|
|
36
|
+
|
|
37
|
+
Published. The tag is what cut it: `git tag v0.5.3 && git push origin main
|
|
38
|
+
v0.5.3`, and the publish job watches for exactly that.
|
|
39
|
+
|
|
40
|
+
This is the release that carries the rename, because 0.5.2 - the release the
|
|
41
|
+
rename was written for - never reached the registry.
|
|
42
|
+
|
|
43
|
+
### Changed
|
|
44
|
+
|
|
45
|
+
- **The repository is now `pham-canh/opencode-harness-kit`**, renamed from
|
|
46
|
+
`pham-canh/space-harness`. The npm package has been
|
|
47
|
+
`opencode-harness-kit` since 0.4.0, and the README's title with it; the
|
|
48
|
+
repository was the last one still carrying the old name, and a reader who
|
|
49
|
+
installed the package and followed the link landed somewhere that did not
|
|
50
|
+
look like the thing they had installed. GitHub redirects the old URL, so
|
|
51
|
+
the links that already exist keep working.
|
|
52
|
+
|
|
53
|
+
The npm page showed the old name until this release, and the reason is
|
|
54
|
+
worth writing down, because nothing about the rename explains it: the page
|
|
55
|
+
reads the `repository` and `homepage` out of the manifest of the LATEST
|
|
56
|
+
PUBLISHED version, and the latest published version was 0.5.1, whose
|
|
57
|
+
manifest was written before the rename. The manifest names the new
|
|
58
|
+
repository now, and this is the release that carries it.
|
|
59
|
+
|
|
60
|
+
### Fixed
|
|
61
|
+
|
|
62
|
+
- **The publish job now checks the OIDC claim against `package.json` before
|
|
63
|
+
it publishes.** npm matches the trusted publisher's repository against the
|
|
64
|
+
claim, and the claim names the repository the workflow ran in - so a
|
|
65
|
+
renamed repository stops publishing, with an exchange error that does not
|
|
66
|
+
name the field that was wrong. The gate says which side is which and
|
|
67
|
+
prints the command that fixes it.
|
|
68
|
+
|
|
69
|
+
- **The installer no longer says "installed" when it replaced nothing.**
|
|
70
|
+
`existsSync` follows a symlink, so a link whose target no longer exists
|
|
71
|
+
reads as "not there" - and the branch that replaces what the installer put
|
|
72
|
+
there was never reached. `symlinkSync` then threw EEXIST over the link
|
|
73
|
+
that was already there, the warning was printed where nobody reads it, and
|
|
74
|
+
the install carried on. After the rename that is exactly the state a kit
|
|
75
|
+
installed from the old path was in: the skills, the prompts and GLOBAL.md
|
|
76
|
+
kept pointing at a directory that had moved, and `verify` then found
|
|
77
|
+
seventeen broken paths. `lstat` does not follow the link, and it is what
|
|
78
|
+
decides whether there is something to replace.
|
|
79
|
+
|
|
80
|
+
### Removed
|
|
81
|
+
|
|
82
|
+
- **The coverage report is no longer committed.** 131 files of it, four
|
|
83
|
+
megabytes, changing on every run, and nobody reads a diff of it. It is
|
|
84
|
+
ignored now and produced on demand.
|
|
85
|
+
|
|
86
|
+
- **The two shell scripts are gone from the repository**: the demo setup
|
|
87
|
+
helper, which pointed at copies of two tools that live one directory up
|
|
88
|
+
with their own history, and the take script that filmed the published
|
|
89
|
+
demo, which is a shell file with one machine's paths in it and goes stale
|
|
90
|
+
the moment the thing it films changes. The recording guide is on the
|
|
91
|
+
machine that records.
|
|
92
|
+
|
|
93
|
+
## [0.5.1] — 2026-10-21
|
|
94
|
+
|
|
95
|
+
Published. The tag is what cut it: `git tag v0.5.1 && git push origin main
|
|
96
|
+
v0.5.1`, and the publish job watches for exactly that.
|
|
97
|
+
|
|
98
|
+
A patch, and it is a patch because nothing a user does differently is in
|
|
99
|
+
here: the grader was stricter than it said, and five tests were reporting
|
|
100
|
+
green with the behaviour they name broken.
|
|
101
|
+
|
|
102
|
+
### Fixed
|
|
103
|
+
|
|
104
|
+
- **The grader accepted a field the agent added.** The comparison with the
|
|
105
|
+
pristine subject looked at the keys the original had, so an agent that made
|
|
106
|
+
the right change and also added a field of its own was graded correct. The
|
|
107
|
+
added key is a change like any other, and the grader says so.
|
|
108
|
+
|
|
109
|
+
- **`isExcluded` did not normalise the path.** The same file is written
|
|
110
|
+
several ways - "./dist/index.js", "dist/index.js", "src/../dist/index.js" -
|
|
111
|
+
and the check only caught one spelling. The grader was saved by also
|
|
112
|
+
checking the resolved path; the exported helper was not, and it is the
|
|
113
|
+
helper a caller would reach for.
|
|
114
|
+
|
|
115
|
+
### Test suite
|
|
116
|
+
|
|
117
|
+
Five tests could not fail, and each was found by breaking the code on
|
|
118
|
+
purpose and watching the test stay green:
|
|
119
|
+
|
|
120
|
+
- **"a many-file change that reordered the list" did not test reordering.**
|
|
121
|
+
The fixture put the appended entry first, so a different check caught it
|
|
122
|
+
and the assertion matched both reasons. The comparison with the original -
|
|
123
|
+
the check that matters - was never exercised. There are two tests now.
|
|
124
|
+
- **"the pruner changes nothing when there is no profile" could not fail.**
|
|
125
|
+
The prunable window was empty, and ten DISTINCT outputs all survive
|
|
126
|
+
anyway. It uses repeated outputs now, with a control case.
|
|
127
|
+
- **"a pruner that throws does not break the turn" never reached the
|
|
128
|
+
catch.** A string is iterable in JavaScript, so it yielded characters and
|
|
129
|
+
never threw.
|
|
130
|
+
- **The measurement test did not assert that the measurement measured
|
|
131
|
+
anything.** It ran behind a gate that can never fail and only matched the
|
|
132
|
+
header, so a pruner that measured nothing passed.
|
|
133
|
+
- **The python-fallback test never removed the sqlite3 CLI**, so the
|
|
134
|
+
fallback it was about never ran on a runner that has it.
|
|
135
|
+
|
|
136
|
+
Also fixed: a lane loop that could iterate zero times; two escape hatches
|
|
137
|
+
that made a test assert nothing at all; a git-root test that used the
|
|
138
|
+
checkout and so failed in a source tarball; seventeen temporary directories
|
|
139
|
+
leaked a run; and a profile index read with a fallback that made a missing
|
|
140
|
+
key mean "no entries" rather than a failure.
|
|
141
|
+
|
|
15
142
|
## [0.5.0] — 2026-10-21
|
|
16
143
|
|
|
17
144
|
A MINOR release, and the reason is the pruner: the rule that decides what
|
|
@@ -1081,20 +1208,26 @@ silent in the worst way:
|
|
|
1081
1208
|
|
|
1082
1209
|
---
|
|
1083
1210
|
|
|
1084
|
-
[Unreleased]: https://github.com/pham-canh/
|
|
1085
|
-
[0.
|
|
1086
|
-
[0.
|
|
1087
|
-
[0.
|
|
1088
|
-
[0.
|
|
1089
|
-
[0.4.
|
|
1090
|
-
[0.
|
|
1091
|
-
[0.
|
|
1092
|
-
[0.3
|
|
1093
|
-
[0.
|
|
1094
|
-
[0.
|
|
1095
|
-
[0.
|
|
1096
|
-
[0.3.
|
|
1097
|
-
[0.3.
|
|
1098
|
-
[0.
|
|
1099
|
-
[0.
|
|
1100
|
-
[0.
|
|
1211
|
+
[Unreleased]: https://github.com/pham-canh/opencode-harness-kit/compare/v0.5.3...HEAD
|
|
1212
|
+
[0.5.3]: https://github.com/pham-canh/opencode-harness-kit/compare/v0.5.2...v0.5.3
|
|
1213
|
+
[0.5.2]: https://github.com/pham-canh/opencode-harness-kit/compare/v0.5.1...v0.5.2
|
|
1214
|
+
[0.5.1]: https://github.com/pham-canh/opencode-harness-kit/compare/v0.5.0...v0.5.1
|
|
1215
|
+
[0.5.0]: https://github.com/pham-canh/opencode-harness-kit/compare/v0.4.6...v0.5.0
|
|
1216
|
+
[0.4.6]: https://github.com/pham-canh/opencode-harness-kit/compare/v0.4.5...v0.4.6
|
|
1217
|
+
[0.4.5]: https://github.com/pham-canh/opencode-harness-kit/compare/v0.4.4...v0.4.5
|
|
1218
|
+
[0.4.4]: https://github.com/pham-canh/opencode-harness-kit/compare/v0.4.3...v0.4.4
|
|
1219
|
+
[0.4.3]: https://github.com/pham-canh/opencode-harness-kit/compare/v0.4.2...v0.4.3
|
|
1220
|
+
[0.4.2]: https://github.com/pham-canh/opencode-harness-kit/compare/v0.4.1...v0.4.2
|
|
1221
|
+
[0.4.1]: https://github.com/pham-canh/opencode-harness-kit/compare/v0.4.0...v0.4.1
|
|
1222
|
+
[0.4.0]: https://github.com/pham-canh/opencode-harness-kit/compare/v0.3.9...v0.4.0
|
|
1223
|
+
[0.3.9]: https://github.com/pham-canh/opencode-harness-kit/compare/v0.3.8...v0.3.9
|
|
1224
|
+
[0.3.8]: https://github.com/pham-canh/opencode-harness-kit/compare/v0.3.7...v0.3.8
|
|
1225
|
+
[0.3.7]: https://github.com/pham-canh/opencode-harness-kit/compare/v0.3.6...v0.3.7
|
|
1226
|
+
[0.3.6]: https://github.com/pham-canh/opencode-harness-kit/compare/v0.3.5...v0.3.6
|
|
1227
|
+
[0.3.5]: https://github.com/pham-canh/opencode-harness-kit/compare/v0.3.4...v0.3.5
|
|
1228
|
+
[0.3.4]: https://github.com/pham-canh/opencode-harness-kit/compare/v0.3.1...v0.3.4
|
|
1229
|
+
[0.3.1]: https://github.com/pham-canh/opencode-harness-kit/compare/v0.3.0...v0.3.1
|
|
1230
|
+
[0.3.0]: https://github.com/pham-canh/opencode-harness-kit/compare/v0.2.1...v0.3.0
|
|
1231
|
+
[0.2.1]: https://github.com/pham-canh/opencode-harness-kit/compare/v0.2.0...v0.2.1
|
|
1232
|
+
[0.2.0]: https://github.com/pham-canh/opencode-harness-kit/compare/v0.1.0...v0.2.0
|
|
1233
|
+
[0.1.0]: https://github.com/pham-canh/opencode-harness-kit/releases/tag/v0.1.0
|
package/CONTRIBUTING.md
CHANGED
|
@@ -69,6 +69,13 @@ config, run the layers individually and say so in the PR.
|
|
|
69
69
|
not part of any check, and a PR does not need it. `npm run measure` does not
|
|
70
70
|
spawn an agent either — it queries the install database read-only.
|
|
71
71
|
|
|
72
|
+
The nightly workflow needs a provider to measure with, and the provider key
|
|
73
|
+
is a **secret** — which a job-level `if` cannot read. The flag is therefore
|
|
74
|
+
a repository **variable**: set `HAS_OPENCODE_KEY` to `true` when
|
|
75
|
+
`OPENCODE_API_KEY` is there, and the job runs. Without the variable the job
|
|
76
|
+
is skipped, not failed, because a schedule that is always red is a schedule
|
|
77
|
+
nobody reads.
|
|
78
|
+
|
|
72
79
|
`eval/models.mjs` spawns real agents too, one line each, against the live
|
|
73
80
|
tier: it is what the routing table in `docs/ROUTING.md` is measured with,
|
|
74
81
|
and it costs a request per model per rep. It is not part of any check
|
|
@@ -132,7 +139,7 @@ repository secret to a granular token scoped to `opencode-harness-kit` with
|
|
|
132
139
|
if it is absent the job uses OIDC.
|
|
133
140
|
|
|
134
141
|
```bash
|
|
135
|
-
gh secret set NPM_TOKEN --repo pham-canh/
|
|
142
|
+
gh secret set NPM_TOKEN --repo pham-canh/opencode-harness-kit
|
|
136
143
|
```
|
|
137
144
|
|
|
138
145
|
> **`gh secret set` takes the secret NAME as its first argument**, and the
|
|
@@ -206,7 +213,7 @@ To set it without the script, the command is:
|
|
|
206
213
|
```bash
|
|
207
214
|
npm trust github opencode-harness-kit \
|
|
208
215
|
--file ci.yml \
|
|
209
|
-
--repo pham-canh/
|
|
216
|
+
--repo pham-canh/opencode-harness-kit \
|
|
210
217
|
--allow-publish --allow-stage-publish -y
|
|
211
218
|
```
|
|
212
219
|
|
|
@@ -215,7 +222,7 @@ The configuration it writes is:
|
|
|
215
222
|
| field | value |
|
|
216
223
|
|---|---|
|
|
217
224
|
| Organization or user | `pham-canh` |
|
|
218
|
-
| Repository | `
|
|
225
|
+
| Repository | `opencode-harness-kit` |
|
|
219
226
|
| Workflow filename | `ci.yml` (the filename only, not a path) |
|
|
220
227
|
| Environment name | **empty** |
|
|
221
228
|
| Allowed actions | `npm publish` and `npm stage publish` |
|
|
@@ -229,7 +236,7 @@ The workflow is the one in `.github/workflows/ci.yml`, and the claims it
|
|
|
229
236
|
presents are printed by the publish job before it tries:
|
|
230
237
|
|
|
231
238
|
```
|
|
232
|
-
repository pham-canh/
|
|
239
|
+
repository pham-canh/opencode-harness-kit
|
|
233
240
|
workflow file ci.yml
|
|
234
241
|
environment (absent) - the trusted publisher must name NO environment
|
|
235
242
|
owner pham-canh
|
package/README.md
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
# opencode-harness-kit
|
|
2
2
|
|
|
3
|
-
[](https://github.com/pham-canh/opencode-harness-kit/actions/workflows/ci.yml)
|
|
4
4
|
[](https://www.npmjs.com/package/opencode-harness-kit)
|
|
5
5
|
|
|
6
6
|
<p align="center">
|
|
@@ -8,9 +8,9 @@
|
|
|
8
8
|
</p>
|
|
9
9
|
|
|
10
10
|
*Four minutes forty-seven seconds, recorded live from a real terminal and
|
|
11
|
-
narrated
|
|
12
|
-
|
|
13
|
-
|
|
11
|
+
narrated — every command on screen is typed and executed while the camera
|
|
12
|
+
runs. The voiceover is in [`video/`](video/), along with what it took to
|
|
13
|
+
record it and the two mistakes that cost the most time.*
|
|
14
14
|
|
|
15
15
|
*It was recorded against an earlier version of the kit: the numbers it names
|
|
16
16
|
are the ones from then, and the ones below are the ones now.*
|
|
@@ -31,7 +31,7 @@ your side — `npx` only downloads the published kit. Prefer git? The same three
|
|
|
31
31
|
steps from a checkout (see [Install](#install)):
|
|
32
32
|
|
|
33
33
|
```bash
|
|
34
|
-
git clone https://github.com/pham-canh/
|
|
34
|
+
git clone https://github.com/pham-canh/opencode-harness-kit.git
|
|
35
35
|
./bin/oc-harness.mjs init
|
|
36
36
|
```
|
|
37
37
|
|
|
@@ -67,6 +67,33 @@ src/ ← no model name anywhere in here
|
|
|
67
67
|
Same plugin source, twelve shipped profiles, twelve different behaviours. The
|
|
68
68
|
test suite asserts that rather than taking it on faith.
|
|
69
69
|
|
|
70
|
+
### How a request travels
|
|
71
|
+
|
|
72
|
+
```mermaid
|
|
73
|
+
flowchart LR
|
|
74
|
+
P[profile JSON<br/>model, effort, fallback, retention] --> C[oc-harness use<br/>compiles the profile]
|
|
75
|
+
C --> CFG[~/.config/opencode/opencode.jsonc<br/>agent.<lane>.model]
|
|
76
|
+
CFG --> PL[the plugin: guard, prune, router, telemetry]
|
|
77
|
+
PL --> LANES[10 lanes<br/>each on the model built for it]
|
|
78
|
+
LANES --> PR[prune: old tool output<br/>becomes a stub]
|
|
79
|
+
LANES --> GD[guard: denied commands<br/>and writes outside the project]
|
|
80
|
+
LANES --> RT[router: a spent route<br/>moves to the next rung]
|
|
81
|
+
LANES --> TL[telemetry: what each lane<br/>actually cost]
|
|
82
|
+
```
|
|
83
|
+
|
|
84
|
+
Four things happen to a request, and they are four separate files:
|
|
85
|
+
|
|
86
|
+
| | |
|
|
87
|
+
|---|---|
|
|
88
|
+
| `src/guard.ts` | refuses the commands the kit never allows, and any write outside the project |
|
|
89
|
+
| `src/prune.ts` | replaces old tool output with a one-line stub, so the window holds the work and not the noise |
|
|
90
|
+
| `src/router.ts` | when a route is spent, moves the lane to the next rung of its own chain |
|
|
91
|
+
| `src/telemetry.ts` | records what each lane actually cost, so `oc-harness lanes` can say which ones earn their place |
|
|
92
|
+
|
|
93
|
+
The model for a lane is in the **config**, not in a prompt and not in a file
|
|
94
|
+
the agent reads. That is the whole design: the answers are data, and a data
|
|
95
|
+
edit is reviewable.
|
|
96
|
+
|
|
70
97
|
## A fleet, not a model
|
|
71
98
|
|
|
72
99
|
The default profile is a **fleet**: ten jobs, ten free models, one session.
|
|
@@ -690,8 +717,8 @@ Without registry access, the checkout path works identically:
|
|
|
690
717
|
|
|
691
718
|
```bash
|
|
692
719
|
pkg install nodejs git
|
|
693
|
-
git clone https://github.com/pham-canh/
|
|
694
|
-
./
|
|
720
|
+
git clone https://github.com/pham-canh/opencode-harness-kit.git
|
|
721
|
+
./opencode-harness-kit/bin/oc-harness.mjs init
|
|
695
722
|
```
|
|
696
723
|
|
|
697
724
|
The kit writes to `$HOME/.config/opencode` like everywhere else, so there is
|
|
@@ -830,7 +857,7 @@ short never grows the history far enough for pruning to matter. So there is
|
|
|
830
857
|
a second suite with one task that touches twelve files and asks a question
|
|
831
858
|
about the code that consumes what it changed — and on it, with the pruner's
|
|
832
859
|
rule fixed (see below), the kit's arm makes the same number of tool calls
|
|
833
|
-
as the arm without the kit and takes 1.2x the wall time.
|
|
860
|
+
as the arm without the kit (47 against 46) and takes 1.2x the wall time.
|
|
834
861
|
|
|
835
862
|
**The pruner used to make the agent re-read.** The first run of that task
|
|
836
863
|
said the kit was much worse: 86 tool calls against 43 and 282,000 input
|
|
@@ -863,6 +890,38 @@ session, so switching happens at the next session, not mid-flight. What a
|
|
|
863
890
|
plugin *can* do is move a lane to the next rung of its chain before the
|
|
864
891
|
session starts.
|
|
865
892
|
|
|
893
|
+
## What is next, and what is not measured yet
|
|
894
|
+
|
|
895
|
+
The claims this project makes are measured, and where a measurement has not
|
|
896
|
+
happened the claim is written down as a gap rather than as a fact. These are
|
|
897
|
+
the open ones.
|
|
898
|
+
|
|
899
|
+
**The pruner's rule was measured on one task.** Keeping the newest copy of
|
|
900
|
+
each distinct tool output instead of a window on recent calls cut the input
|
|
901
|
+
tokens by 69% and the wall time by 60% on the long-task A/B. One task is one
|
|
902
|
+
workload. It needs the same treatment on a repository that is not this one,
|
|
903
|
+
and on a session that runs for hours rather than minutes.
|
|
904
|
+
|
|
905
|
+
**The reviewer's "you never fix" is an instruction, not a boundary.** The
|
|
906
|
+
measurement of it came out even: the kit's reviewer lane and a default agent
|
|
907
|
+
behaved the same on a review task, and a task that says "fix it" overrode
|
|
908
|
+
the lane. The kit's own loop keeps the line by never asking a reviewer to
|
|
909
|
+
fix anything. A guard that cannot be overridden by a task is worth having,
|
|
910
|
+
and that is not built yet.
|
|
911
|
+
|
|
912
|
+
**The nightly A/B needs a provider key on the repository.** The workflow
|
|
913
|
+
exists, it runs, and it is skipped until the key is set — the variable
|
|
914
|
+
`HAS_OPENCODE_KEY` says the secret is there. The numbers in
|
|
915
|
+
[eval/RESULTS.md](eval/RESULTS.md) are the ones taken by hand.
|
|
916
|
+
|
|
917
|
+
**The TypeScript pilot is one public repository.** It is now a real one
|
|
918
|
+
(`typesense-js`, pinned by commit) rather than a private tree, which is what
|
|
919
|
+
made it checkable. Two languages are not a survey.
|
|
920
|
+
|
|
921
|
+
**The lanes are ten because ten models were available.** That is a
|
|
922
|
+
hypothesis about the tier, not a number. `oc-harness lanes` counts the
|
|
923
|
+
requests and says which lanes were never used; cutting them is a data edit.
|
|
924
|
+
|
|
866
925
|
## Development
|
|
867
926
|
|
|
868
927
|
```bash
|
|
@@ -907,9 +966,15 @@ and the measurements behind each choice.
|
|
|
907
966
|
|
|
908
967
|
[`CHANGELOG.md`](CHANGELOG.md) records what each release changed and, for the
|
|
909
968
|
releases before this one, what was broken in it — the silent failures that a
|
|
910
|
-
config could look correct through. [`MIGRATION.md`](MIGRATION.md) covers
|
|
911
|
-
0.2.x → 0.3.0
|
|
912
|
-
telemetry file now means something
|
|
969
|
+
config could look correct through. [`MIGRATION.md`](MIGRATION.md) covers the
|
|
970
|
+
two upgrades that ask something of you: 0.2.x → 0.3.0 (the profile ids
|
|
971
|
+
changed, the config layout changed, the telemetry file now means something
|
|
972
|
+
different) and 0.3.x → 0.4.x (the guard started refusing in code what it used
|
|
973
|
+
to only suggest, and the pre-commit hook started running your project's
|
|
974
|
+
tests instead of the kit's).
|
|
975
|
+
|
|
976
|
+
[`docs/EXIT-CODES.md`](docs/EXIT-CODES.md) lists what every command returns
|
|
977
|
+
when it fails, so a script can tell "the kit says no" from "the kit broke".
|
|
913
978
|
|
|
914
979
|
## Contributing
|
|
915
980
|
|
package/SECURITY.md
CHANGED
|
@@ -12,7 +12,7 @@
|
|
|
12
12
|
## Reporting a vulnerability
|
|
13
13
|
|
|
14
14
|
Please report privately through GitHub's
|
|
15
|
-
[private vulnerability reporting](https://github.com/pham-canh/
|
|
15
|
+
[private vulnerability reporting](https://github.com/pham-canh/opencode-harness-kit/security/advisories/new)
|
|
16
16
|
rather than opening a public issue.
|
|
17
17
|
|
|
18
18
|
Include the affected version, a reproduction, and the expected behaviour.
|
|
@@ -21,7 +21,7 @@ plan within 7 days.
|
|
|
21
21
|
|
|
22
22
|
## Scope
|
|
23
23
|
|
|
24
|
-
`
|
|
24
|
+
`opencode-harness-kit` runs agent subprocesses and reads harness config. Reports
|
|
25
25
|
that are in scope include path traversal when resolving a capability
|
|
26
26
|
profile, command injection through harness config values, and any way a
|
|
27
27
|
harness writes outside its configured working directory.
|
package/bin/oc-harness.mjs
CHANGED
|
@@ -706,19 +706,32 @@ function linkOne(src, dest) {
|
|
|
706
706
|
}
|
|
707
707
|
if (ours) return
|
|
708
708
|
|
|
709
|
-
if (
|
|
709
|
+
if (isLink) {
|
|
710
|
+
// A symlink is a symlink whether or not its target exists, and a
|
|
711
|
+
// DANGLING one is still this installer's to replace.
|
|
712
|
+
//
|
|
713
|
+
// `existsSync` follows the link, so it says false for a link whose
|
|
714
|
+
// source has moved - the repository was renamed, or the kit was
|
|
715
|
+
// reinstalled somewhere else. The old code asked `existsSync` whether
|
|
716
|
+
// the destination was taken, got "no", and then tried to create a link
|
|
717
|
+
// over an existing one: EEXIST, printed as a warning nobody reads, and
|
|
718
|
+
// the install reported success. Every skill, prompt and rule kept
|
|
719
|
+
// pointing at a directory that is no longer there, and `verify` found
|
|
720
|
+
// seventeen broken paths afterwards.
|
|
721
|
+
//
|
|
722
|
+
// The test that decides whether to replace something is `lstat`, which
|
|
723
|
+
// does not follow the link.
|
|
724
|
+
rmSync(dest, { force: true })
|
|
725
|
+
} else if (existsSync(dest)) {
|
|
710
726
|
// Replace only what this installer put there. The old code treated "not
|
|
711
727
|
// already our link" as "safe to delete" and called `rmSync` with
|
|
712
728
|
// `recursive: false`, which succeeds on a plain file: a hand-written
|
|
713
729
|
// agent was deleted and replaced by a symlink without a word. A real
|
|
714
730
|
// directory is left alone too -- unlinking it is not this installer's
|
|
715
731
|
// call to make.
|
|
716
|
-
|
|
717
|
-
|
|
718
|
-
|
|
719
|
-
return
|
|
720
|
-
}
|
|
721
|
-
rmSync(dest, { force: true })
|
|
732
|
+
const kind = lstatSync(dest).isDirectory() ? "directory" : "file"
|
|
733
|
+
console.log(` ${yellow("!")} ${stamp(`left alone: ${dest} is a real ${kind} and was not ours to replace`)}`)
|
|
734
|
+
return
|
|
722
735
|
}
|
|
723
736
|
|
|
724
737
|
try {
|
package/eval/RESULTS.md
CHANGED
|
@@ -146,6 +146,115 @@ That is the mechanism the whole kit rests on, and it is the one thing the
|
|
|
146
146
|
correlation in Results 1 and 2 could not show. The kit does not shrink what it
|
|
147
147
|
costs; it shrinks what the agent drags along behind it.
|
|
148
148
|
|
|
149
|
+
## The A/B on this repository, 10 reps per arm
|
|
150
|
+
|
|
151
|
+
node eval/prepare.mjs --repo . --ref <sha> --name self
|
|
152
|
+
node eval/ab.mjs --target <scratch>/self --suite eval/tasks-self.json --reps 10
|
|
153
|
+
|
|
154
|
+
**The subject is this repository at a pinned commit**, so the run is
|
|
155
|
+
reproducible: `git clone && git checkout <sha>` and the same command. The
|
|
156
|
+
previous suite ran against a private repository, which meant a result nobody
|
|
157
|
+
else could check.
|
|
158
|
+
|
|
159
|
+
Six tasks, ten reps per arm, 120 agent runs: four lookups and two edits. The
|
|
160
|
+
lookups are graded on the `file:line` the agent cited, read out of the
|
|
161
|
+
snapshot; the edits are graded on the file after the run.
|
|
162
|
+
|
|
163
|
+
**Both arms ran on the same model**, `opencode/minimax-m3.1-flash-free`, and
|
|
164
|
+
the report prints it. The kit-off arm used to run with an empty config, so
|
|
165
|
+
opencode picked its own default: not the model the kit-on arm was on, and
|
|
166
|
+
not a model that answered - four kit-off runs hit the 600-second ceiling
|
|
167
|
+
with no output at all, and a timed-out run has zero tools and zero context,
|
|
168
|
+
so it entered the medians as a fast, empty, wrong answer.
|
|
169
|
+
|
|
170
|
+
| metric (median) | kit on | kit off |
|
|
171
|
+
| --- | --- | --- |
|
|
172
|
+
| **correct** | **54/60** | **55/60** |
|
|
173
|
+
| tool calls | 12.5 | 8 |
|
|
174
|
+
| read calls | 3 | 2 |
|
|
175
|
+
| narrow reads | 66.7% | 66.7% |
|
|
176
|
+
| grep/glob calls | 3 | 2 |
|
|
177
|
+
| steps | 9 | 6 |
|
|
178
|
+
| input tokens | 56,205 | 10,540 |
|
|
179
|
+
| final context | 39,928 | 12,718 |
|
|
180
|
+
| wall seconds | 43 | 21 |
|
|
181
|
+
| paths written outside the target | 0 | 0 |
|
|
182
|
+
|
|
183
|
+
| per task | kit on | kit off |
|
|
184
|
+
| --- | --- | --- |
|
|
185
|
+
| `read-control` | 10/10 | 10/10 |
|
|
186
|
+
| `read-multi-denylist` | 8/10 | 8/10 |
|
|
187
|
+
| `read-multi-gate` | 6/10 | 7/10 |
|
|
188
|
+
| `read-two-files` | 10/10 | 10/10 |
|
|
189
|
+
| `edit-rename` | 10/10 | 10/10 |
|
|
190
|
+
| `edit-multi` | 10/10 | 10/10 |
|
|
191
|
+
|
|
192
|
+
### What this says
|
|
193
|
+
|
|
194
|
+
**On correctness, the kit is a tie.** 54 against 55, on sixty runs a side.
|
|
195
|
+
That is inside the noise of a single task flipping, and it is the opposite
|
|
196
|
+
of what the previous version of this file said. A second run of the same
|
|
197
|
+
120 runs - same subject, same suite, same model, a different day of model
|
|
198
|
+
traffic - gave **55/60 against 54/60**, the other way. Two measurements
|
|
199
|
+
put the tie on either side of zero, which is what a tie looks like from
|
|
200
|
+
both ends.
|
|
201
|
+
|
|
202
|
+
**On everything else, the kit is a cost.** It spends **5.3x the input
|
|
203
|
+
tokens** and takes **twice as long**, and it makes **more** tool calls.
|
|
204
|
+
Those are the honest numbers.
|
|
205
|
+
|
|
206
|
+
**The previous conclusion was the grader's fault, and it is worth writing
|
|
207
|
+
down.** The earlier run said 52/60 against 59/59 - the kit looked clearly
|
|
208
|
+
worse - and the reason was that the grader could not read a citation
|
|
209
|
+
written as two markdown fields:
|
|
210
|
+
|
|
211
|
+
**File:** `src/profile.js`
|
|
212
|
+
**Line:** 105
|
|
213
|
+
|
|
214
|
+
The old pattern stopped at the newline, so the file and the number were
|
|
215
|
+
never paired, the citation was lost, and the answer was scored wrong. The
|
|
216
|
+
kit's own instructions tell the agent to cite a file and a line, and the
|
|
217
|
+
shape the kit produces is the shape the grader could not read. The kit-off
|
|
218
|
+
arm had no such instructions, answered in prose, and was graded by a
|
|
219
|
+
pattern that handled prose.
|
|
220
|
+
|
|
221
|
+
Re-scoring the EARLIER run with the fixed grader - the same 120 agent runs,
|
|
222
|
+
the same answers, only the grader changed - gives **34/40 against 30/40** on
|
|
223
|
+
the four lookup tasks, the kit ahead. A harness that measures the harness
|
|
224
|
+
is worse than one that measures nothing.
|
|
225
|
+
|
|
226
|
+
### What this does not say
|
|
227
|
+
|
|
228
|
+
It does not say the kit is not worth it. It says the kit's value is **not in
|
|
229
|
+
short lookups**, and this suite only contains short lookups and two small
|
|
230
|
+
edits. The value it does have is in two places this suite cannot see:
|
|
231
|
+
|
|
232
|
+
- **Long sessions.** The pruner removes **65-80% of the context** across real
|
|
233
|
+
sessions, depending on which sessions it is measured over
|
|
234
|
+
(`npm run measure:prune`), and a task that fits in one read never gets
|
|
235
|
+
near the point where pruning matters. The long-task A/B below is the
|
|
236
|
+
measurement of that.
|
|
237
|
+
- **Process.** A `reviewer` that reviews and does not edit, a `verify` that
|
|
238
|
+
runs the checks and does not fix them, a `space-finish` that hands the
|
|
239
|
+
publish decision to a human. Those are behaviours, not accuracy, and a
|
|
240
|
+
grader that counts correct answers cannot see them. The process section
|
|
241
|
+
below measures what it can - and what it found is that the lane and the
|
|
242
|
+
default agent behave the same, and that a task saying "fix it" overrides
|
|
243
|
+
the lane.
|
|
244
|
+
|
|
245
|
+
The honest position is a trade, and it is now a trade with a number on it:
|
|
246
|
+
**the kit costs about five times the input tokens on a small task, gets the
|
|
247
|
+
answer right about as often, and buys process discipline and an 80% smaller
|
|
248
|
+
context on a long one.** Whether that is worth it depends on whether the
|
|
249
|
+
work is long. A repo of one-file lookups should not install this.
|
|
250
|
+
|
|
251
|
+
The narrow-read share is worth one more line. The kit's instructions say
|
|
252
|
+
*locate before you read*, and the share of reads that were bounded was
|
|
253
|
+
**the same on both sides, 66.7%** - the rule is followed, and on a
|
|
254
|
+
one-file question following it is waste either way. A fleet that routes
|
|
255
|
+
each job to the model built for it is the other half of what the kit
|
|
256
|
+
offers, and this suite uses one model on both sides, so it cannot see that.
|
|
257
|
+
|
|
149
258
|
## The A/B on TypeScript, against a PUBLIC subject
|
|
150
259
|
|
|
151
260
|
node eval/prepare.mjs --clone https://github.com/typesense/typesense-js \
|
|
@@ -221,11 +330,11 @@ pristine subject file by file, and the answer is graded by citation.
|
|
|
221
330
|
| metric (median of 2 runs per arm) | kit on | kit off |
|
|
222
331
|
| --- | --- | --- |
|
|
223
332
|
| **correct** | **2/2** | **2/2** |
|
|
224
|
-
| tool calls |
|
|
225
|
-
| read calls |
|
|
226
|
-
| input tokens |
|
|
227
|
-
| final context |
|
|
228
|
-
| wall seconds |
|
|
333
|
+
| tool calls | 47 | 46 |
|
|
334
|
+
| read calls | 12 | 18 |
|
|
335
|
+
| input tokens | 78,280 | 37,171 |
|
|
336
|
+
| final context | 68,794 | 42,732 |
|
|
337
|
+
| wall seconds | 121.4 | 104.7 |
|
|
229
338
|
|
|
230
339
|
### What the long task found, and what it cost to fix
|
|
231
340
|
|
|
@@ -259,8 +368,8 @@ The context is bigger and the work is not repeated. That is the trade the
|
|
|
259
368
|
kit was making badly, and it was making it without saying.
|
|
260
369
|
|
|
261
370
|
**What is left is the instructions, not the pruner.** With the rule on, the
|
|
262
|
-
kit's arm makes the
|
|
263
|
-
|
|
371
|
+
kit's arm makes the SAME number of tool calls as the arm without the kit -
|
|
372
|
+
47 against 46, against 67.5 before the fix - and takes 1.2x the wall time.
|
|
264
373
|
The remaining cost is the fixed overhead of carrying the kit's rules, and
|
|
265
374
|
that is the trade the README describes.
|
|
266
375
|
|
package/eval/verify.mjs
CHANGED
|
@@ -180,14 +180,31 @@ export function resolveInTarget(target, ref) {
|
|
|
180
180
|
* @returns {boolean}
|
|
181
181
|
*/
|
|
182
182
|
export function isExcluded(file, exclude = []) {
|
|
183
|
-
|
|
183
|
+
// The path is NORMALISED, because the same file is written several ways:
|
|
184
|
+
// "./dist/index.js", "dist/index.js" and "src/../dist/index.js" are one
|
|
185
|
+
// file, and a check that only catches one spelling is not a check.
|
|
186
|
+
const norm = String(file ?? "")
|
|
187
|
+
.replace(/\\/g, "/")
|
|
188
|
+
.replace(/^\.\//, "")
|
|
189
|
+
.replace(/^\.\//g, "")
|
|
190
|
+
const parts = []
|
|
191
|
+
for (const seg of norm.split("/")) {
|
|
192
|
+
if (!seg || seg === ".") continue
|
|
193
|
+
if (seg === "..") {
|
|
194
|
+
parts.pop()
|
|
195
|
+
continue
|
|
196
|
+
}
|
|
197
|
+
parts.push(seg)
|
|
198
|
+
}
|
|
199
|
+
const clean = parts.join("/")
|
|
200
|
+
const base = parts[parts.length - 1] ?? ""
|
|
184
201
|
return exclude.some((x) => {
|
|
185
202
|
if (!x) return false
|
|
186
|
-
if (x.endsWith("/")) return
|
|
203
|
+
if (x.endsWith("/")) return clean.startsWith(x)
|
|
187
204
|
// A file NAME excludes that file wherever it is: the suite is excluded as
|
|
188
205
|
// "tasks-self.json", and the citation is "eval/tasks-self.json".
|
|
189
|
-
if (
|
|
190
|
-
return
|
|
206
|
+
if (clean === x || base === x) return true
|
|
207
|
+
return clean.startsWith(`${x}/`)
|
|
191
208
|
})
|
|
192
209
|
}
|
|
193
210
|
|
|
@@ -353,6 +370,13 @@ export function gradeMutationMany(runDir, spec, { pristine } = {}) {
|
|
|
353
370
|
return { ok: false, reason: `${file}: ${key} changed, and only ${spec.field} was asked for` }
|
|
354
371
|
}
|
|
355
372
|
}
|
|
373
|
+
// And a key the agent ADDED is a change too. Only the keys the original
|
|
374
|
+
// had were compared, so an agent that appended the right entry and also
|
|
375
|
+
// added a field of its own was graded correct.
|
|
376
|
+
for (const key of Object.keys(data)) {
|
|
377
|
+
if (key in orig) continue
|
|
378
|
+
return { ok: false, reason: `${file}: ${key} was added, and only ${spec.field} was asked for` }
|
|
379
|
+
}
|
|
356
380
|
if (JSON.stringify(arr) !== JSON.stringify(origArr)) changed.push(file)
|
|
357
381
|
}
|
|
358
382
|
return { ok: true, changed }
|
package/lanes.schema.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"$schema": "http://json-schema.org/draft-07/schema#",
|
|
3
|
-
"$id": "https://github.com/pham-canh/
|
|
3
|
+
"$id": "https://github.com/pham-canh/opencode-harness-kit/lanes.schema.json",
|
|
4
4
|
"title": "OpenCode harness lane table",
|
|
5
5
|
"description": "The jobs a fleet can hand out. A lane says what it is for and nothing about models: a profile says which lanes it enables and which model serves each one. The lane table is the single list of lane names in the kit; `src/profile.js` reads it, `src/config.js` compiles it into the generated config, and `profiles/schema.json` must name the same lanes.",
|
|
6
6
|
"type": "object",
|
package/package.json
CHANGED
|
@@ -1,15 +1,15 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "opencode-harness-kit",
|
|
3
|
-
"version": "0.5.
|
|
3
|
+
"version": "0.5.3",
|
|
4
4
|
"description": "Harness kit for OpenCode: capability profiles, a fleet of job lanes with one free model each, role-based reasoning effort, context pruning, and measured baselines.",
|
|
5
5
|
"license": "MIT",
|
|
6
|
-
"homepage": "https://github.com/pham-canh/
|
|
6
|
+
"homepage": "https://github.com/pham-canh/opencode-harness-kit#readme",
|
|
7
7
|
"repository": {
|
|
8
8
|
"type": "git",
|
|
9
|
-
"url": "git+https://github.com/pham-canh/
|
|
9
|
+
"url": "git+https://github.com/pham-canh/opencode-harness-kit.git"
|
|
10
10
|
},
|
|
11
11
|
"bugs": {
|
|
12
|
-
"url": "https://github.com/pham-canh/
|
|
12
|
+
"url": "https://github.com/pham-canh/opencode-harness-kit/issues"
|
|
13
13
|
},
|
|
14
14
|
"type": "module",
|
|
15
15
|
"main": "./dist/index.js",
|
package/profiles/schema.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"$schema": "http://json-schema.org/draft-07/schema#",
|
|
3
|
-
"$id": "https://github.com/pham-canh/
|
|
3
|
+
"$id": "https://github.com/pham-canh/opencode-harness-kit/profiles/schema.json",
|
|
4
4
|
"title": "OpenCode harness profile",
|
|
5
5
|
"description": "Everything that changes when you change model. Plugins read this file and contain no model names of their own.",
|
|
6
6
|
"type": "object",
|