harness-everything 0.23.0 → 0.24.0

package/.claude-plugin/marketplace.json

@@ -10,7 +10,7 @@
       "name": "harness-everything",
       "source": "./",
       "description": "Harness OS: an orchestrated agent operating system — tier routing, circuit breaker guardrails, verification gates, WAL session memory, and self-evolving workspace rules.",
-      "version": "0.23.0",
+      "version": "0.24.0",
       "author": {
         "name": "Miya Daniel"
       },

package/.claude-plugin/plugin.json

@@ -1,7 +1,7 @@
 {
   "name": "harness-everything",
   "description": "Harness OS: an orchestrated agent operating system — tier routing, circuit breaker guardrails, verification gates, WAL session memory, and self-evolving workspace rules.",
-  "version": "0.23.0",
+  "version": "0.24.0",
   "author": {
     "name": "Miya Daniel"
   },

package/CHANGELOG.md

@@ -2,6 +2,19 @@
 
 All notable changes to this project are documented in this file.
 
+## [0.24.0](https://github.com/dyphn1/Harness-everything/compare/v0.23.0...v0.24.0) (2026-09-23)
+
+### Features
+
+* **router:** apply calibrated system one acceptance thresholds ([61841a2](https://github.com/dyphn1/Harness-everything/commit/61841a28afc0c8ba0163625b28974962f0cf78f7)), closes [#233](https://github.com/dyphn1/Harness-everything/issues/233)
+* **router:** collect local owner prompts for system one training ([c84d543](https://github.com/dyphn1/Harness-everything/commit/c84d5433e2d152fd57618b24691c75a50e1fc3c8)), closes [#233](https://github.com/dyphn1/Harness-everything/issues/233)
+* **router:** label system one training prompts with a checked llm ([5056a9a](https://github.com/dyphn1/Harness-everything/commit/5056a9a7abc2820dc6b9eedf8266bd39082afd8f)), closes [#233](https://github.com/dyphn1/Harness-everything/issues/233)
+* **router:** train and calibrate the system one tier scorer ([cc26296](https://github.com/dyphn1/Harness-everything/commit/cc26296cd128cc75a74a5fbe2d3c8637068781b1)), closes [#233](https://github.com/dyphn1/Harness-everything/issues/233)
+
+### Bug Fixes
+
+* **router:** record why system one labeling batches fail ([c6dd6f6](https://github.com/dyphn1/Harness-everything/commit/c6dd6f61b19020a95ccf0fbef30ac9d5113a69f4)), closes [#233](https://github.com/dyphn1/Harness-everything/issues/233)
+
 ## [0.23.0](https://github.com/dyphn1/Harness-everything/compare/v0.22.1...v0.23.0) (2026-09-23)
 
 ### Features

package/benchmarks/results/system-one/labeler-holdout-check-2026-09-23/README.md

@@ -0,0 +1,55 @@
+# System One: LLM labeler check against the owner-reviewed holdout (2026-09-23)
+
+Evidence layer: **one-time agreement check** between the training-data labeler and the
+owner's gold. It measures label quality only. It is neither model evidence nor routing
+evidence. Issue [#233](https://github.com/dyphn1/Harness-everything/issues/233). The labeling
+procedure is defined in [system-one-training.md](../../../../docs/system-one-training.md).
+
+## Command
+
+```bash
+node scripts/system-one-label.js check-holdout \
+  --out benchmarks/results/system-one/labeler-holdout-check-2026-09-23/report.json --concurrency 3
+```
+
+[`report.json`](report.json) is the unmodified output. It lists case IDs with gold and
+predicted labels only; prompt text is not included.
+
+- **Labeler**: headless `claude -p` with Sonnet (`claude-sonnet-5`). Settings, hooks, plugins,
+  tools and session persistence are disabled, and output follows a JSON schema.
+- **Rules**: the gold section of `docs/system-one-corpus.md`, with SHA-256 prefix
+  `87871ae710981284`.
+- **Batching**: 100 prompts per batch.
+
+## Result
+
+| Metric | Value |
+| --- | --- |
+| Cases | 215 (0 missing, 0 failed) |
+| Agreement with the owner's gold | **86.0%** (the gate is at least 80%) |
+| Recall: tier1 / tier2 / tier3 / null | 87.2% / 84.5% / 88.2% / 85.0% |
+
+Disagreements, shown as gold → labeler:
+
+| Pair | Cases |
+| --- | --- |
+| tier2 → tier3 | 8 |
+| tier2 → tier1 | 7 |
+| tier3 → tier2 | 5 |
+| tier1 → tier2 | 4 |
+| tier1 → null | 2 |
+| null → tier2 | 2 |
+| tier3 → tier1 | 1 |
+| null → tier1 | 1 |
+
+All but one disagreement lies between adjacent tiers or involves `null`. The gate passes, so
+the training data can be labeled with these instructions unchanged. The instructions were not
+tuned on this holdout.
+
+## Limitations
+
+- An agreement of about 86% implies roughly 14% label noise in the training data. That noise
+  bounds what a model trained on these labels can reach against the owner's gold.
+- The owner's spot-check of the training labels measures the noise on the training
+  distribution itself.
+- This is one run with one model; run-to-run variance was not measured.