harness-everything 0.20.7 → 0.22.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/.claude-plugin/marketplace.json +1 -1
- package/.claude-plugin/plugin.json +1 -1
- package/CHANGELOG.md +26 -0
- package/benchmarks/fixtures/system-one-routing.json +549 -0
- package/benchmarks/results/system-one/cua-s1-forms-cpu-2026-09-23/README.md +62 -0
- package/benchmarks/results/system-one/cua-s1-forms-cpu-2026-09-23/report.json +856 -0
- package/benchmarks/results/system-one/cua-s1-forms-resident-cpu-2026-09-23/README.md +74 -0
- package/benchmarks/results/system-one/cua-s1-forms-resident-cpu-2026-09-23/report.json +856 -0
- package/ci/fixtures/system-one-provider.cjs +13 -0
- package/ci/fixtures/system-one-resident-stub.py +28 -0
- package/ci/mechanism-packaging-bytecode.test.js +61 -0
- package/ci/mechanism-system-one-contract.test.js +48 -0
- package/ci/mechanism-system-one-evaluation.test.js +87 -0
- package/ci/mechanism-system-one-install.test.js +106 -0
- package/ci/mechanism-system-one-provider.test.js +63 -0
- package/ci/mechanism-system-one-resident.test.js +185 -0
- package/ci/mechanism-system-one-router.test.js +101 -0
- package/ci/system-one-adapter-test.py +222 -0
- package/docs/platform-capabilities.md +14 -0
- package/docs/system-one-routing.md +271 -0
- package/harness-everything/SKILL.md +1 -1
- package/harness-everything/scripts/system-one/contract.js +45 -0
- package/harness-everything/scripts/system-one/cua_adapter.py +353 -0
- package/harness-everything/scripts/system-one/evaluate.js +98 -0
- package/harness-everything/scripts/system-one/install.js +161 -0
- package/harness-everything/scripts/system-one/provider.js +79 -0
- package/harness-everything/scripts/system-one/resident.js +176 -0
- package/harness-everything/scripts/system-one/router.js +37 -0
- package/harness-everything/scripts/tier-router.js +16 -0
- package/opencode-plugin/plugin.json +1 -1
- package/package.json +2 -1
- package/plugins/harness-everything/.codex-plugin/plugin.json +1 -1
- package/plugins/harness-everything/plugin.json +1 -1
- package/plugins/harness-everything/scripts/lib/packaging-noise.js +6 -0
- package/plugins/harness-everything/scripts/lib/skills.js +2 -0
- package/plugins/harness-everything/skills/harness-everything/SKILL.md +1 -1
- package/plugins/harness-everything/skills/harness-everything/scripts/system-one/contract.js +45 -0
- package/plugins/harness-everything/skills/harness-everything/scripts/system-one/cua_adapter.py +353 -0
- package/plugins/harness-everything/skills/harness-everything/scripts/system-one/evaluate.js +98 -0
- package/plugins/harness-everything/skills/harness-everything/scripts/system-one/install.js +161 -0
- package/plugins/harness-everything/skills/harness-everything/scripts/system-one/provider.js +79 -0
- package/plugins/harness-everything/skills/harness-everything/scripts/system-one/resident.js +176 -0
- package/plugins/harness-everything/skills/harness-everything/scripts/system-one/router.js +37 -0
- package/plugins/harness-everything/skills/harness-everything/scripts/tier-router.js +16 -0
- package/sbom.cdx.json +7 -7
- package/scripts/build-openai-submission.js +2 -0
- package/scripts/evaluate-system-one.js +66 -0
- package/scripts/lib/packaging-noise.js +6 -0
- package/scripts/lib/skills.js +2 -0
- package/scripts/sync-openai-plugin.js +2 -1
|
@@ -10,7 +10,7 @@
|
|
|
10
10
|
"name": "harness-everything",
|
|
11
11
|
"source": "./",
|
|
12
12
|
"description": "Harness OS: an orchestrated agent operating system — tier routing, circuit breaker guardrails, verification gates, WAL session memory, and self-evolving workspace rules.",
|
|
13
|
-
"version": "0.
|
|
13
|
+
"version": "0.22.0",
|
|
14
14
|
"author": {
|
|
15
15
|
"name": "Miya Daniel"
|
|
16
16
|
},
|
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "harness-everything",
|
|
3
3
|
"description": "Harness OS: an orchestrated agent operating system — tier routing, circuit breaker guardrails, verification gates, WAL session memory, and self-evolving workspace rules.",
|
|
4
|
-
"version": "0.
|
|
4
|
+
"version": "0.22.0",
|
|
5
5
|
"author": {
|
|
6
6
|
"name": "Miya Daniel"
|
|
7
7
|
},
|
package/CHANGELOG.md
CHANGED
|
@@ -2,6 +2,32 @@
|
|
|
2
2
|
|
|
3
3
|
All notable changes to this project are documented in this file.
|
|
4
4
|
|
|
5
|
+
## [0.22.0](https://github.com/dyphn1/Harness-everything/compare/v0.21.0...v0.22.0) (2026-09-23)
|
|
6
|
+
|
|
7
|
+
### Features
|
|
8
|
+
|
|
9
|
+
* **router:** add resident system one provider ([eee8cb7](https://github.com/dyphn1/Harness-everything/commit/eee8cb763d2ed65722a2578c83700b02bdc3e668)), closes [#233](https://github.com/dyphn1/Harness-everything/issues/233)
|
|
10
|
+
|
|
11
|
+
### Bug Fixes
|
|
12
|
+
|
|
13
|
+
* **distribution:** keep python bytecode out of every package ([58135ae](https://github.com/dyphn1/Harness-everything/commit/58135aef92089d731eb44d55efb0ae77ea4d64dd)), closes [#233](https://github.com/dyphn1/Harness-everything/issues/233)
|
|
14
|
+
* **router:** harden resident system one server from adversarial review ([ee5ab0f](https://github.com/dyphn1/Harness-everything/commit/ee5ab0fec9f03808f12a88e66b769b210803baa2)), closes [#233](https://github.com/dyphn1/Harness-everything/issues/233)
|
|
15
|
+
* **router:** replace the whole windows dacl for resident state ([e59af73](https://github.com/dyphn1/Harness-everything/commit/e59af735de1337b3bcb10cbca05500ab6b03b64c)), closes [#233](https://github.com/dyphn1/Harness-everything/issues/233)
|
|
16
|
+
|
|
17
|
+
## [0.21.0](https://github.com/dyphn1/Harness-everything/compare/v0.20.7...v0.21.0) (2026-09-23)
|
|
18
|
+
|
|
19
|
+
### Features
|
|
20
|
+
|
|
21
|
+
* **router:** add opt-in system one installer with real cpu evidence ([87eae6c](https://github.com/dyphn1/Harness-everything/commit/87eae6c5ce6c217ab4c3baf5786e8fd14ec1e8ad)), closes [#233](https://github.com/dyphn1/Harness-everything/issues/233)
|
|
22
|
+
* **router:** add optional local CUA-S1 scoring adapter ([5bb2840](https://github.com/dyphn1/Harness-everything/commit/5bb2840dd1cf6ac27b9ed540e00b70246905d277))
|
|
23
|
+
* **router:** add paired System One evaluation and rollout gates ([6b0ee09](https://github.com/dyphn1/Harness-everything/commit/6b0ee0953acf2b5b36c67c82f9221fd30d3b3223))
|
|
24
|
+
* **router:** define bounded System One scoring contract ([b00923b](https://github.com/dyphn1/Harness-everything/commit/b00923b7fa51594d97334911e631d75c83f882af))
|
|
25
|
+
* **router:** integrate opt-in System One tier scoring ([f5bd079](https://github.com/dyphn1/Harness-everything/commit/f5bd07985aa90bf2b5245c8eceaa3666538e192e))
|
|
26
|
+
|
|
27
|
+
### Bug Fixes
|
|
28
|
+
|
|
29
|
+
* **router:** enforce system one structural floor and full evaluation evidence ([5949467](https://github.com/dyphn1/Harness-everything/commit/5949467aa2d9c4cee8beeabe721b53ab82516220)), closes [#234](https://github.com/dyphn1/Harness-everything/issues/234) [#233](https://github.com/dyphn1/Harness-everything/issues/233)
|
|
30
|
+
|
|
5
31
|
## [0.20.7](https://github.com/dyphn1/Harness-everything/compare/v0.20.6...v0.20.7) (2026-09-22)
|
|
6
32
|
|
|
7
33
|
### Bug Fixes
|
|
@@ -0,0 +1,549 @@
|
|
|
1
|
+
{
|
|
2
|
+
"schemaVersion": 1,
|
|
3
|
+
"cases": [
|
|
4
|
+
{
|
|
5
|
+
"id": "typo-en",
|
|
6
|
+
"family": "typo",
|
|
7
|
+
"language": "en",
|
|
8
|
+
"split": "holdout",
|
|
9
|
+
"source": "issue-233: illustrative authored fixture, not independently reviewed gold",
|
|
10
|
+
"reviewed": false,
|
|
11
|
+
"request": {
|
|
12
|
+
"schemaVersion": 1,
|
|
13
|
+
"task": "tier",
|
|
14
|
+
"context": "Fix one spelling typo in README.",
|
|
15
|
+
"options": [
|
|
16
|
+
{
|
|
17
|
+
"id": "tier1",
|
|
18
|
+
"text": "A small bounded edit or operation with clear scope."
|
|
19
|
+
},
|
|
20
|
+
{
|
|
21
|
+
"id": "tier2",
|
|
22
|
+
"text": "Implementation or bug fix needing iterative testing."
|
|
23
|
+
},
|
|
24
|
+
{
|
|
25
|
+
"id": "tier3",
|
|
26
|
+
"text": "Architecture, repository-wide change or multi-stage engineering."
|
|
27
|
+
},
|
|
28
|
+
{
|
|
29
|
+
"id": "unclassified",
|
|
30
|
+
"text": "Unclear intent, insufficient context or outside these categories."
|
|
31
|
+
}
|
|
32
|
+
],
|
|
33
|
+
"catalogHash": "b625bd9390f1d163f8fbace0b9c4768166bfc613aed04d87b2246b2c6e966d40",
|
|
34
|
+
"requestHash": "842f482c252d3669f7b83d52f8756e52d15b9b3090c5ec3dff1e816fae19934c"
|
|
35
|
+
},
|
|
36
|
+
"gold": "tier1"
|
|
37
|
+
},
|
|
38
|
+
{
|
|
39
|
+
"id": "typo-zh",
|
|
40
|
+
"family": "typo",
|
|
41
|
+
"language": "zh-TW",
|
|
42
|
+
"split": "holdout",
|
|
43
|
+
"source": "issue-233: illustrative authored fixture, not independently reviewed gold",
|
|
44
|
+
"reviewed": false,
|
|
45
|
+
"request": {
|
|
46
|
+
"schemaVersion": 1,
|
|
47
|
+
"task": "tier",
|
|
48
|
+
"context": "修正 README 中的一個錯字。",
|
|
49
|
+
"options": [
|
|
50
|
+
{
|
|
51
|
+
"id": "tier1",
|
|
52
|
+
"text": "A small bounded edit or operation with clear scope."
|
|
53
|
+
},
|
|
54
|
+
{
|
|
55
|
+
"id": "tier2",
|
|
56
|
+
"text": "Implementation or bug fix needing iterative testing."
|
|
57
|
+
},
|
|
58
|
+
{
|
|
59
|
+
"id": "tier3",
|
|
60
|
+
"text": "Architecture, repository-wide change or multi-stage engineering."
|
|
61
|
+
},
|
|
62
|
+
{
|
|
63
|
+
"id": "unclassified",
|
|
64
|
+
"text": "Unclear intent, insufficient context or outside these categories."
|
|
65
|
+
}
|
|
66
|
+
],
|
|
67
|
+
"catalogHash": "b625bd9390f1d163f8fbace0b9c4768166bfc613aed04d87b2246b2c6e966d40",
|
|
68
|
+
"requestHash": "843caa4dbd9433bab15453e696fc5974e249c5a6f2aa557e318bcd62e864a95d"
|
|
69
|
+
},
|
|
70
|
+
"gold": "tier1"
|
|
71
|
+
},
|
|
72
|
+
{
|
|
73
|
+
"id": "bug-en",
|
|
74
|
+
"family": "bug",
|
|
75
|
+
"language": "en",
|
|
76
|
+
"split": "holdout",
|
|
77
|
+
"source": "issue-233: illustrative authored fixture, not independently reviewed gold",
|
|
78
|
+
"reviewed": false,
|
|
79
|
+
"request": {
|
|
80
|
+
"schemaVersion": 1,
|
|
81
|
+
"task": "tier",
|
|
82
|
+
"context": "Fix the parser returning a wrong value for empty input and add regression tests.",
|
|
83
|
+
"options": [
|
|
84
|
+
{
|
|
85
|
+
"id": "tier1",
|
|
86
|
+
"text": "A small bounded edit or operation with clear scope."
|
|
87
|
+
},
|
|
88
|
+
{
|
|
89
|
+
"id": "tier2",
|
|
90
|
+
"text": "Implementation or bug fix needing iterative testing."
|
|
91
|
+
},
|
|
92
|
+
{
|
|
93
|
+
"id": "tier3",
|
|
94
|
+
"text": "Architecture, repository-wide change or multi-stage engineering."
|
|
95
|
+
},
|
|
96
|
+
{
|
|
97
|
+
"id": "unclassified",
|
|
98
|
+
"text": "Unclear intent, insufficient context or outside these categories."
|
|
99
|
+
}
|
|
100
|
+
],
|
|
101
|
+
"catalogHash": "b625bd9390f1d163f8fbace0b9c4768166bfc613aed04d87b2246b2c6e966d40",
|
|
102
|
+
"requestHash": "ff85c15ad4847929336548982924eab97a672376a0657dfd777f4fa78d794c0d"
|
|
103
|
+
},
|
|
104
|
+
"gold": "tier2"
|
|
105
|
+
},
|
|
106
|
+
{
|
|
107
|
+
"id": "bug-zh",
|
|
108
|
+
"family": "bug",
|
|
109
|
+
"language": "zh-TW",
|
|
110
|
+
"split": "holdout",
|
|
111
|
+
"source": "issue-233: illustrative authored fixture, not independently reviewed gold",
|
|
112
|
+
"reviewed": false,
|
|
113
|
+
"request": {
|
|
114
|
+
"schemaVersion": 1,
|
|
115
|
+
"task": "tier",
|
|
116
|
+
"context": "修正解析器對空輸入回傳錯誤數值,補回歸測試。",
|
|
117
|
+
"options": [
|
|
118
|
+
{
|
|
119
|
+
"id": "tier1",
|
|
120
|
+
"text": "A small bounded edit or operation with clear scope."
|
|
121
|
+
},
|
|
122
|
+
{
|
|
123
|
+
"id": "tier2",
|
|
124
|
+
"text": "Implementation or bug fix needing iterative testing."
|
|
125
|
+
},
|
|
126
|
+
{
|
|
127
|
+
"id": "tier3",
|
|
128
|
+
"text": "Architecture, repository-wide change or multi-stage engineering."
|
|
129
|
+
},
|
|
130
|
+
{
|
|
131
|
+
"id": "unclassified",
|
|
132
|
+
"text": "Unclear intent, insufficient context or outside these categories."
|
|
133
|
+
}
|
|
134
|
+
],
|
|
135
|
+
"catalogHash": "b625bd9390f1d163f8fbace0b9c4768166bfc613aed04d87b2246b2c6e966d40",
|
|
136
|
+
"requestHash": "c237e19662ed1f707f5049d7ceda63d917d5c7ba676487f93355aa8319089294"
|
|
137
|
+
},
|
|
138
|
+
"gold": "tier2"
|
|
139
|
+
},
|
|
140
|
+
{
|
|
141
|
+
"id": "design-en",
|
|
142
|
+
"family": "design",
|
|
143
|
+
"language": "en",
|
|
144
|
+
"split": "holdout",
|
|
145
|
+
"source": "issue-233: illustrative authored fixture, not independently reviewed gold",
|
|
146
|
+
"reviewed": false,
|
|
147
|
+
"request": {
|
|
148
|
+
"schemaVersion": 1,
|
|
149
|
+
"task": "tier",
|
|
150
|
+
"context": "Redesign the router architecture and migrate every platform.",
|
|
151
|
+
"options": [
|
|
152
|
+
{
|
|
153
|
+
"id": "tier1",
|
|
154
|
+
"text": "A small bounded edit or operation with clear scope."
|
|
155
|
+
},
|
|
156
|
+
{
|
|
157
|
+
"id": "tier2",
|
|
158
|
+
"text": "Implementation or bug fix needing iterative testing."
|
|
159
|
+
},
|
|
160
|
+
{
|
|
161
|
+
"id": "tier3",
|
|
162
|
+
"text": "Architecture, repository-wide change or multi-stage engineering."
|
|
163
|
+
},
|
|
164
|
+
{
|
|
165
|
+
"id": "unclassified",
|
|
166
|
+
"text": "Unclear intent, insufficient context or outside these categories."
|
|
167
|
+
}
|
|
168
|
+
],
|
|
169
|
+
"catalogHash": "b625bd9390f1d163f8fbace0b9c4768166bfc613aed04d87b2246b2c6e966d40",
|
|
170
|
+
"requestHash": "77dfb5b40a6d485aed079a6767f768e5c737a87d09c03380f7090cf03b18956a"
|
|
171
|
+
},
|
|
172
|
+
"gold": "tier3"
|
|
173
|
+
},
|
|
174
|
+
{
|
|
175
|
+
"id": "design-zh",
|
|
176
|
+
"family": "design",
|
|
177
|
+
"language": "zh-TW",
|
|
178
|
+
"split": "holdout",
|
|
179
|
+
"source": "issue-233: illustrative authored fixture, not independently reviewed gold",
|
|
180
|
+
"reviewed": false,
|
|
181
|
+
"request": {
|
|
182
|
+
"schemaVersion": 1,
|
|
183
|
+
"task": "tier",
|
|
184
|
+
"context": "重新設計 router 架構並遷移所有平台。",
|
|
185
|
+
"options": [
|
|
186
|
+
{
|
|
187
|
+
"id": "tier1",
|
|
188
|
+
"text": "A small bounded edit or operation with clear scope."
|
|
189
|
+
},
|
|
190
|
+
{
|
|
191
|
+
"id": "tier2",
|
|
192
|
+
"text": "Implementation or bug fix needing iterative testing."
|
|
193
|
+
},
|
|
194
|
+
{
|
|
195
|
+
"id": "tier3",
|
|
196
|
+
"text": "Architecture, repository-wide change or multi-stage engineering."
|
|
197
|
+
},
|
|
198
|
+
{
|
|
199
|
+
"id": "unclassified",
|
|
200
|
+
"text": "Unclear intent, insufficient context or outside these categories."
|
|
201
|
+
}
|
|
202
|
+
],
|
|
203
|
+
"catalogHash": "b625bd9390f1d163f8fbace0b9c4768166bfc613aed04d87b2246b2c6e966d40",
|
|
204
|
+
"requestHash": "0014fdf2fa5c27bfbe67609fe4cc3237ee74ff147b7037a8ee9c5dd533eade3c"
|
|
205
|
+
},
|
|
206
|
+
"gold": "tier3"
|
|
207
|
+
},
|
|
208
|
+
{
|
|
209
|
+
"id": "ambiguous-en",
|
|
210
|
+
"family": "ambiguous",
|
|
211
|
+
"language": "en",
|
|
212
|
+
"split": "holdout",
|
|
213
|
+
"source": "issue-233: illustrative authored fixture, not independently reviewed gold",
|
|
214
|
+
"reviewed": false,
|
|
215
|
+
"request": {
|
|
216
|
+
"schemaVersion": 1,
|
|
217
|
+
"task": "tier",
|
|
218
|
+
"context": "Take a look at this.",
|
|
219
|
+
"options": [
|
|
220
|
+
{
|
|
221
|
+
"id": "tier1",
|
|
222
|
+
"text": "A small bounded edit or operation with clear scope."
|
|
223
|
+
},
|
|
224
|
+
{
|
|
225
|
+
"id": "tier2",
|
|
226
|
+
"text": "Implementation or bug fix needing iterative testing."
|
|
227
|
+
},
|
|
228
|
+
{
|
|
229
|
+
"id": "tier3",
|
|
230
|
+
"text": "Architecture, repository-wide change or multi-stage engineering."
|
|
231
|
+
},
|
|
232
|
+
{
|
|
233
|
+
"id": "unclassified",
|
|
234
|
+
"text": "Unclear intent, insufficient context or outside these categories."
|
|
235
|
+
}
|
|
236
|
+
],
|
|
237
|
+
"catalogHash": "b625bd9390f1d163f8fbace0b9c4768166bfc613aed04d87b2246b2c6e966d40",
|
|
238
|
+
"requestHash": "e77726d7ac34b0f56be6394f3401dd24dcab5bb577ca3e1076dd7c75bd8439d7"
|
|
239
|
+
},
|
|
240
|
+
"gold": null
|
|
241
|
+
},
|
|
242
|
+
{
|
|
243
|
+
"id": "ambiguous-zh",
|
|
244
|
+
"family": "ambiguous",
|
|
245
|
+
"language": "zh-TW",
|
|
246
|
+
"split": "holdout",
|
|
247
|
+
"source": "issue-233: illustrative authored fixture, not independently reviewed gold",
|
|
248
|
+
"reviewed": false,
|
|
249
|
+
"request": {
|
|
250
|
+
"schemaVersion": 1,
|
|
251
|
+
"task": "tier",
|
|
252
|
+
"context": "看看這個。",
|
|
253
|
+
"options": [
|
|
254
|
+
{
|
|
255
|
+
"id": "tier1",
|
|
256
|
+
"text": "A small bounded edit or operation with clear scope."
|
|
257
|
+
},
|
|
258
|
+
{
|
|
259
|
+
"id": "tier2",
|
|
260
|
+
"text": "Implementation or bug fix needing iterative testing."
|
|
261
|
+
},
|
|
262
|
+
{
|
|
263
|
+
"id": "tier3",
|
|
264
|
+
"text": "Architecture, repository-wide change or multi-stage engineering."
|
|
265
|
+
},
|
|
266
|
+
{
|
|
267
|
+
"id": "unclassified",
|
|
268
|
+
"text": "Unclear intent, insufficient context or outside these categories."
|
|
269
|
+
}
|
|
270
|
+
],
|
|
271
|
+
"catalogHash": "b625bd9390f1d163f8fbace0b9c4768166bfc613aed04d87b2246b2c6e966d40",
|
|
272
|
+
"requestHash": "a4a3f89a4a7d42fb5729796ce0091028a824b124509fb8afa7b4936044db0321"
|
|
273
|
+
},
|
|
274
|
+
"gold": null
|
|
275
|
+
},
|
|
276
|
+
{
|
|
277
|
+
"id": "negation-en",
|
|
278
|
+
"family": "negation",
|
|
279
|
+
"language": "en",
|
|
280
|
+
"split": "holdout",
|
|
281
|
+
"source": "issue-233: illustrative authored fixture, not independently reviewed gold",
|
|
282
|
+
"reviewed": false,
|
|
283
|
+
"request": {
|
|
284
|
+
"schemaVersion": 1,
|
|
285
|
+
"task": "tier",
|
|
286
|
+
"context": "Do not refactor the architecture; only fix a README typo.",
|
|
287
|
+
"options": [
|
|
288
|
+
{
|
|
289
|
+
"id": "tier1",
|
|
290
|
+
"text": "A small bounded edit or operation with clear scope."
|
|
291
|
+
},
|
|
292
|
+
{
|
|
293
|
+
"id": "tier2",
|
|
294
|
+
"text": "Implementation or bug fix needing iterative testing."
|
|
295
|
+
},
|
|
296
|
+
{
|
|
297
|
+
"id": "tier3",
|
|
298
|
+
"text": "Architecture, repository-wide change or multi-stage engineering."
|
|
299
|
+
},
|
|
300
|
+
{
|
|
301
|
+
"id": "unclassified",
|
|
302
|
+
"text": "Unclear intent, insufficient context or outside these categories."
|
|
303
|
+
}
|
|
304
|
+
],
|
|
305
|
+
"catalogHash": "b625bd9390f1d163f8fbace0b9c4768166bfc613aed04d87b2246b2c6e966d40",
|
|
306
|
+
"requestHash": "49f4f2cfa209244c514b84c2f7e4b19eebd370448c85b8d59706687e4210884c"
|
|
307
|
+
},
|
|
308
|
+
"gold": "tier1"
|
|
309
|
+
},
|
|
310
|
+
{
|
|
311
|
+
"id": "negation-zh",
|
|
312
|
+
"family": "negation",
|
|
313
|
+
"language": "zh-TW",
|
|
314
|
+
"split": "holdout",
|
|
315
|
+
"source": "issue-233: illustrative authored fixture, not independently reviewed gold",
|
|
316
|
+
"reviewed": false,
|
|
317
|
+
"request": {
|
|
318
|
+
"schemaVersion": 1,
|
|
319
|
+
"task": "tier",
|
|
320
|
+
"context": "不要重構架構,只修改 README 的一個錯字。",
|
|
321
|
+
"options": [
|
|
322
|
+
{
|
|
323
|
+
"id": "tier1",
|
|
324
|
+
"text": "A small bounded edit or operation with clear scope."
|
|
325
|
+
},
|
|
326
|
+
{
|
|
327
|
+
"id": "tier2",
|
|
328
|
+
"text": "Implementation or bug fix needing iterative testing."
|
|
329
|
+
},
|
|
330
|
+
{
|
|
331
|
+
"id": "tier3",
|
|
332
|
+
"text": "Architecture, repository-wide change or multi-stage engineering."
|
|
333
|
+
},
|
|
334
|
+
{
|
|
335
|
+
"id": "unclassified",
|
|
336
|
+
"text": "Unclear intent, insufficient context or outside these categories."
|
|
337
|
+
}
|
|
338
|
+
],
|
|
339
|
+
"catalogHash": "b625bd9390f1d163f8fbace0b9c4768166bfc613aed04d87b2246b2c6e966d40",
|
|
340
|
+
"requestHash": "bfa207d9ca040dc37efca66d9c952f19e42dc203ef4063dbb543f5676fbdd00d"
|
|
341
|
+
},
|
|
342
|
+
"gold": "tier1"
|
|
343
|
+
},
|
|
344
|
+
{
|
|
345
|
+
"id": "mixed-en",
|
|
346
|
+
"family": "mixed",
|
|
347
|
+
"language": "en",
|
|
348
|
+
"split": "holdout",
|
|
349
|
+
"source": "issue-233: illustrative authored fixture, not independently reviewed gold",
|
|
350
|
+
"reviewed": false,
|
|
351
|
+
"request": {
|
|
352
|
+
"schemaVersion": 1,
|
|
353
|
+
"task": "tier",
|
|
354
|
+
"context": "Add a parser, migrate its consumers, update docs, then benchmark all platforms.",
|
|
355
|
+
"options": [
|
|
356
|
+
{
|
|
357
|
+
"id": "tier1",
|
|
358
|
+
"text": "A small bounded edit or operation with clear scope."
|
|
359
|
+
},
|
|
360
|
+
{
|
|
361
|
+
"id": "tier2",
|
|
362
|
+
"text": "Implementation or bug fix needing iterative testing."
|
|
363
|
+
},
|
|
364
|
+
{
|
|
365
|
+
"id": "tier3",
|
|
366
|
+
"text": "Architecture, repository-wide change or multi-stage engineering."
|
|
367
|
+
},
|
|
368
|
+
{
|
|
369
|
+
"id": "unclassified",
|
|
370
|
+
"text": "Unclear intent, insufficient context or outside these categories."
|
|
371
|
+
}
|
|
372
|
+
],
|
|
373
|
+
"catalogHash": "b625bd9390f1d163f8fbace0b9c4768166bfc613aed04d87b2246b2c6e966d40",
|
|
374
|
+
"requestHash": "0a0dfc43e4466aada82da6fc06690dfba91b166cbffd56df4ce82a37aa024a08"
|
|
375
|
+
},
|
|
376
|
+
"gold": "tier3"
|
|
377
|
+
},
|
|
378
|
+
{
|
|
379
|
+
"id": "mixed-zh",
|
|
380
|
+
"family": "mixed",
|
|
381
|
+
"language": "zh-TW",
|
|
382
|
+
"split": "holdout",
|
|
383
|
+
"source": "issue-233: illustrative authored fixture, not independently reviewed gold",
|
|
384
|
+
"reviewed": false,
|
|
385
|
+
"request": {
|
|
386
|
+
"schemaVersion": 1,
|
|
387
|
+
"task": "tier",
|
|
388
|
+
"context": "新增解析器、遷移呼叫端、更新文件,再評測所有平台。",
|
|
389
|
+
"options": [
|
|
390
|
+
{
|
|
391
|
+
"id": "tier1",
|
|
392
|
+
"text": "A small bounded edit or operation with clear scope."
|
|
393
|
+
},
|
|
394
|
+
{
|
|
395
|
+
"id": "tier2",
|
|
396
|
+
"text": "Implementation or bug fix needing iterative testing."
|
|
397
|
+
},
|
|
398
|
+
{
|
|
399
|
+
"id": "tier3",
|
|
400
|
+
"text": "Architecture, repository-wide change or multi-stage engineering."
|
|
401
|
+
},
|
|
402
|
+
{
|
|
403
|
+
"id": "unclassified",
|
|
404
|
+
"text": "Unclear intent, insufficient context or outside these categories."
|
|
405
|
+
}
|
|
406
|
+
],
|
|
407
|
+
"catalogHash": "b625bd9390f1d163f8fbace0b9c4768166bfc613aed04d87b2246b2c6e966d40",
|
|
408
|
+
"requestHash": "554f608a3e5c783251e587ab72fa1e4e47501574c5f871256aa04f8e3086cbb1"
|
|
409
|
+
},
|
|
410
|
+
"gold": "tier3"
|
|
411
|
+
},
|
|
412
|
+
{
|
|
413
|
+
"id": "explicit-en",
|
|
414
|
+
"family": "explicit",
|
|
415
|
+
"language": "en",
|
|
416
|
+
"split": "holdout",
|
|
417
|
+
"source": "issue-233: illustrative authored fixture, not independently reviewed gold",
|
|
418
|
+
"reviewed": false,
|
|
419
|
+
"request": {
|
|
420
|
+
"schemaVersion": 1,
|
|
421
|
+
"task": "tier",
|
|
422
|
+
"context": "Use fable-staged to design and implement a new workflow.",
|
|
423
|
+
"options": [
|
|
424
|
+
{
|
|
425
|
+
"id": "tier1",
|
|
426
|
+
"text": "A small bounded edit or operation with clear scope."
|
|
427
|
+
},
|
|
428
|
+
{
|
|
429
|
+
"id": "tier2",
|
|
430
|
+
"text": "Implementation or bug fix needing iterative testing."
|
|
431
|
+
},
|
|
432
|
+
{
|
|
433
|
+
"id": "tier3",
|
|
434
|
+
"text": "Architecture, repository-wide change or multi-stage engineering."
|
|
435
|
+
},
|
|
436
|
+
{
|
|
437
|
+
"id": "unclassified",
|
|
438
|
+
"text": "Unclear intent, insufficient context or outside these categories."
|
|
439
|
+
}
|
|
440
|
+
],
|
|
441
|
+
"catalogHash": "b625bd9390f1d163f8fbace0b9c4768166bfc613aed04d87b2246b2c6e966d40",
|
|
442
|
+
"requestHash": "a6c7a4dc6bbfe3daadfc6ba2e66592ce4344a59067e5ca2ab5f90738c72f2e05"
|
|
443
|
+
},
|
|
444
|
+
"gold": "tier3"
|
|
445
|
+
},
|
|
446
|
+
{
|
|
447
|
+
"id": "outside-en",
|
|
448
|
+
"family": "outside",
|
|
449
|
+
"language": "en",
|
|
450
|
+
"split": "holdout",
|
|
451
|
+
"source": "issue-233: illustrative authored fixture, not independently reviewed gold",
|
|
452
|
+
"reviewed": false,
|
|
453
|
+
"request": {
|
|
454
|
+
"schemaVersion": 1,
|
|
455
|
+
"task": "tier",
|
|
456
|
+
"context": "What is your favorite food?",
|
|
457
|
+
"options": [
|
|
458
|
+
{
|
|
459
|
+
"id": "tier1",
|
|
460
|
+
"text": "A small bounded edit or operation with clear scope."
|
|
461
|
+
},
|
|
462
|
+
{
|
|
463
|
+
"id": "tier2",
|
|
464
|
+
"text": "Implementation or bug fix needing iterative testing."
|
|
465
|
+
},
|
|
466
|
+
{
|
|
467
|
+
"id": "tier3",
|
|
468
|
+
"text": "Architecture, repository-wide change or multi-stage engineering."
|
|
469
|
+
},
|
|
470
|
+
{
|
|
471
|
+
"id": "unclassified",
|
|
472
|
+
"text": "Unclear intent, insufficient context or outside these categories."
|
|
473
|
+
}
|
|
474
|
+
],
|
|
475
|
+
"catalogHash": "b625bd9390f1d163f8fbace0b9c4768166bfc613aed04d87b2246b2c6e966d40",
|
|
476
|
+
"requestHash": "ad70746c3a6c8f94add479be391c1a0f0bce704ae28b7a22f0ae49f6705325eb"
|
|
477
|
+
},
|
|
478
|
+
"gold": null
|
|
479
|
+
},
|
|
480
|
+
{
|
|
481
|
+
"id": "outside-zh",
|
|
482
|
+
"family": "outside",
|
|
483
|
+
"language": "zh-TW",
|
|
484
|
+
"split": "holdout",
|
|
485
|
+
"source": "issue-233: illustrative authored fixture, not independently reviewed gold",
|
|
486
|
+
"reviewed": false,
|
|
487
|
+
"request": {
|
|
488
|
+
"schemaVersion": 1,
|
|
489
|
+
"task": "tier",
|
|
490
|
+
"context": "你喜歡吃什麼?",
|
|
491
|
+
"options": [
|
|
492
|
+
{
|
|
493
|
+
"id": "tier1",
|
|
494
|
+
"text": "A small bounded edit or operation with clear scope."
|
|
495
|
+
},
|
|
496
|
+
{
|
|
497
|
+
"id": "tier2",
|
|
498
|
+
"text": "Implementation or bug fix needing iterative testing."
|
|
499
|
+
},
|
|
500
|
+
{
|
|
501
|
+
"id": "tier3",
|
|
502
|
+
"text": "Architecture, repository-wide change or multi-stage engineering."
|
|
503
|
+
},
|
|
504
|
+
{
|
|
505
|
+
"id": "unclassified",
|
|
506
|
+
"text": "Unclear intent, insufficient context or outside these categories."
|
|
507
|
+
}
|
|
508
|
+
],
|
|
509
|
+
"catalogHash": "b625bd9390f1d163f8fbace0b9c4768166bfc613aed04d87b2246b2c6e966d40",
|
|
510
|
+
"requestHash": "ca3c6372387fc48ddd19a24bea15fc1a2c0a590f43ec715af10646ecf3a41206"
|
|
511
|
+
},
|
|
512
|
+
"gold": null
|
|
513
|
+
},
|
|
514
|
+
{
|
|
515
|
+
"id": "long-en",
|
|
516
|
+
"family": "long",
|
|
517
|
+
"language": "en",
|
|
518
|
+
"split": "holdout",
|
|
519
|
+
"source": "issue-233: illustrative authored fixture, not independently reviewed gold",
|
|
520
|
+
"reviewed": false,
|
|
521
|
+
"request": {
|
|
522
|
+
"schemaVersion": 1,
|
|
523
|
+
"task": "tier",
|
|
524
|
+
"context": "The following is historical context. The following is historical context. The following is historical context. The following is historical context. The following is historical context. The following is historical context. The following is historical context. The following is historical context. The following is historical context. The following is historical context. The following is historical context. The following is historical context. Only fix a spelling typo in the README.",
|
|
525
|
+
"options": [
|
|
526
|
+
{
|
|
527
|
+
"id": "tier1",
|
|
528
|
+
"text": "A small bounded edit or operation with clear scope."
|
|
529
|
+
},
|
|
530
|
+
{
|
|
531
|
+
"id": "tier2",
|
|
532
|
+
"text": "Implementation or bug fix needing iterative testing."
|
|
533
|
+
},
|
|
534
|
+
{
|
|
535
|
+
"id": "tier3",
|
|
536
|
+
"text": "Architecture, repository-wide change or multi-stage engineering."
|
|
537
|
+
},
|
|
538
|
+
{
|
|
539
|
+
"id": "unclassified",
|
|
540
|
+
"text": "Unclear intent, insufficient context or outside these categories."
|
|
541
|
+
}
|
|
542
|
+
],
|
|
543
|
+
"catalogHash": "b625bd9390f1d163f8fbace0b9c4768166bfc613aed04d87b2246b2c6e966d40",
|
|
544
|
+
"requestHash": "2146aced0d7a197b77ad47f548c506657dd667291117942210dfb3479b2fce04"
|
|
545
|
+
},
|
|
546
|
+
"gold": "tier1"
|
|
547
|
+
}
|
|
548
|
+
]
|
|
549
|
+
}
|
|
@@ -0,0 +1,62 @@
|
|
|
1
|
+
# System One: real CUA-S1-FORMS CPU inference (2026-09-23)
|
|
2
|
+
|
|
3
|
+
Evidence layer: **local real-checkpoint inference and paired offline evaluation**.
|
|
4
|
+
It is not live-host evidence, not a reviewed holdout, and not a Harness routing
|
|
5
|
+
quality claim. Issue [#233](https://github.com/dyphn1/Harness-everything/issues/233).
|
|
6
|
+
|
|
7
|
+
## Setup
|
|
8
|
+
|
|
9
|
+
- Installed with `node harness-everything/scripts/system-one/install.js` (default
|
|
10
|
+
directory `~/.agents/harness-everything/system-one`); exit 0, `verification.ok: true`.
|
|
11
|
+
A second run reported both checkpoint files as `verified-existing`.
|
|
12
|
+
- Provenance probe: CPython 3.13.6, torch `2.14.0+cpu`, distribution `cua-s1 0.0.0`,
|
|
13
|
+
PEP 610 source revision `b7f7e2d8714609853a29c7d049140bc46aec0954` → `pinned`.
|
|
14
|
+
- Checkpoint: `cua-ai/cua-s1-forms` at Hugging Face revision
|
|
15
|
+
`f54adbf447f4ca6ec259f529ee3f2e3e09f8cc71`; weights SHA-256
|
|
16
|
+
`05954c1c…6ddc` (LFS oid), sidecar SHA-256 `62d31e2f…50ca` (git blob
|
|
17
|
+
`d8f8426a…` re-hashed); declared domain `forms-v1`.
|
|
18
|
+
- Host: Intel Core Ultra 5 225H, Windows 11 (win32 10.0.26200 x64), Node v24.18.0,
|
|
19
|
+
one torch thread.
|
|
20
|
+
|
|
21
|
+
## Command
|
|
22
|
+
|
|
23
|
+
```bash
|
|
24
|
+
node scripts/evaluate-system-one.js benchmarks/fixtures/system-one-routing.json \
|
|
25
|
+
~/.agents/harness-everything/system-one/manifest.json \
|
|
26
|
+
benchmarks/results/system-one/cua-s1-forms-cpu-2026-09-23/report.json
|
|
27
|
+
```
|
|
28
|
+
|
|
29
|
+
[`report.json`](report.json) is the unmodified output (16 holdout cases × 2 runs, 173 s wall).
|
|
30
|
+
|
|
31
|
+
## Results
|
|
32
|
+
|
|
33
|
+
| Metric | Lexical baseline | CUA-S1-FORMS |
|
|
34
|
+
| --- | --- | --- |
|
|
35
|
+
| Accuracy | 0.625 | 0.25 (null-gold cases only) |
|
|
36
|
+
| Macro-F1 | 0.572 | 0.10 |
|
|
37
|
+
| Coverage | 0.5625 | 0 |
|
|
38
|
+
|
|
39
|
+
- Every scored decision was `abstain/domain-mismatch`; the 225-byte case was
|
|
40
|
+
`provider-exit` (context over the checkpoint's 224-byte limit, no truncation).
|
|
41
|
+
- Repeatability: agreement 1.0 across full score vectors.
|
|
42
|
+
- Latency: cold p95 6415 ms (32 cold samples; every call starts Python and imports torch).
|
|
43
|
+
No warm samples exist.
|
|
44
|
+
- Gates: `repeatability` and `sourceProvenance` pass; all others fail or are pending;
|
|
45
|
+
`rolloutReady: false`.
|
|
46
|
+
|
|
47
|
+
Diagnostic only (not a gate): if the domain check is ignored, raw top-1 would match
|
|
48
|
+
gold on 6/15 scored cases. Several of the wrong answers had very high confidence
|
|
49
|
+
(0.994, 0.996). This confirms the documented model-suitability boundary: the forms
|
|
50
|
+
checkpoint must not drive Harness routing, and the `domain-mismatch` abstention
|
|
51
|
+
is working as intended.
|
|
52
|
+
|
|
53
|
+
Kernel router check (same host): `shadow` and `prefer` with this manifest print a
|
|
54
|
+
`SYSTEM ONE` diagnostic with `domain-mismatch`/`applied: false`, and the emitted
|
|
55
|
+
router workflow plan is byte-identical to `off`. One shadow prompt took ≈7.2 s wall,
|
|
56
|
+
so enabling shadow in a live host adds roughly 5–7 s per prompt with this one-shot bridge.
|
|
57
|
+
|
|
58
|
+
## Limitations
|
|
59
|
+
|
|
60
|
+
The seed corpus is 16 unreviewed mechanism cases. There is no warm/persistent server,
|
|
61
|
+
no policy or live-host evidence, and no Harness-trained checkpoint. None of the
|
|
62
|
+
Phase 4 rollout gates are satisfied.
|