harness-everything 0.22.1 → 0.23.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/.claude-plugin/marketplace.json +1 -1
- package/.claude-plugin/plugin.json +1 -1
- package/CHANGELOG.md +6 -0
- package/benchmarks/fixtures/system-one-holdout-draft.json +2039 -0
- package/benchmarks/fixtures/system-one-holdout-reviews.json +1587 -0
- package/benchmarks/fixtures/system-one-holdout.json +7315 -0
- package/benchmarks/results/system-one/lexical-baseline-reviewed-holdout-2026-09-23/README.md +47 -0
- package/benchmarks/results/system-one/lexical-baseline-reviewed-holdout-2026-09-23/report.json +5222 -0
- package/ci/mechanism-system-one-corpus.test.js +154 -0
- package/docs/system-one-corpus.md +103 -0
- package/docs/system-one-routing.md +2 -1
- package/opencode-plugin/plugin.json +1 -1
- package/package.json +1 -1
- package/plugins/harness-everything/.codex-plugin/plugin.json +1 -1
- package/plugins/harness-everything/plugin.json +1 -1
- package/sbom.cdx.json +7 -7
- package/scripts/system-one-corpus.js +99 -0
|
@@ -0,0 +1,47 @@
|
|
|
1
|
+
# System One: lexical baseline on the reviewed tier holdout (2026-09-23)
|
|
2
|
+
|
|
3
|
+
Evidence layer: **deterministic evaluation of the current lexical router against
|
|
4
|
+
owner-reviewed gold**. No model was run: the manifest path does not exist, so
|
|
5
|
+
every model decision is `provider-config` and model coverage is 0 by design.
|
|
6
|
+
Issue [#233](https://github.com/dyphn1/Harness-everything/issues/233).
|
|
7
|
+
|
|
8
|
+
## Command
|
|
9
|
+
|
|
10
|
+
```bash
|
|
11
|
+
node scripts/evaluate-system-one.js benchmarks/fixtures/system-one-holdout.json \
|
|
12
|
+
/nonexistent/manifest.json \
|
|
13
|
+
benchmarks/results/system-one/lexical-baseline-reviewed-holdout-2026-09-23/report.json
|
|
14
|
+
```
|
|
15
|
+
|
|
16
|
+
[`report.json`](report.json) is the unmodified output. The holdout contains 215
|
|
17
|
+
owner-reviewed cases and is described in
|
|
18
|
+
[system-one-corpus.md](../../../../docs/system-one-corpus.md).
|
|
19
|
+
|
|
20
|
+
## Lexical baseline
|
|
21
|
+
|
|
22
|
+
| Metric | Value |
|
|
23
|
+
| --- | --- |
|
|
24
|
+
| Accuracy (abstention counted as `null`) | 28.8% |
|
|
25
|
+
| Macro-F1 over tier1, tier2, tier3, null | 0.254 |
|
|
26
|
+
| Coverage (a tier was chosen) | 40.5% |
|
|
27
|
+
| Precision of chosen tiers | 49.4% |
|
|
28
|
+
|
|
29
|
+
Gold versus lexical prediction (rows are gold):
|
|
30
|
+
|
|
31
|
+
| Gold \ lexical | tier1 | tier2 | tier3 | unclassified |
|
|
32
|
+
| --- | --- | --- | --- | --- |
|
|
33
|
+
| tier1 (47) | 1 | 6 | 3 | 37 |
|
|
34
|
+
| tier2 (97) | 0 | 28 | 10 | 59 |
|
|
35
|
+
| tier3 (51) | 0 | 24 | 14 | 13 |
|
|
36
|
+
| null (20) | 0 | 0 | 1 | 19 |
|
|
37
|
+
|
|
38
|
+
The keyword router leaves most prompts unclassified and almost never selects
|
|
39
|
+
`tier1`. `reviewedHoldout` passes. A `harness-routing-v1` checkpoint must reach
|
|
40
|
+
at least this macro-F1 (0.254) and also meet accepted precision of at least 98%
|
|
41
|
+
and coverage of at least 80% before any default change.
|
|
42
|
+
|
|
43
|
+
## Limitations
|
|
44
|
+
|
|
45
|
+
This is one annotator's gold on 215 synthetic or de-identified prompts, with no
|
|
46
|
+
inter-annotator agreement measured. It measures the lexical tier decision only,
|
|
47
|
+
not the policy assembler's final workflow plan.
|