harness-everything 0.22.1 → 0.24.0

package/docs/system-one-corpus.md

@@ -0,0 +1,103 @@
+# System One holdout corpus: labeling and review
+
+This document defines how the reviewed tier holdout for
+[System One](system-one-routing.md) is written, labeled, reviewed and
+assembled (issue [#233](https://github.com/dyphn1/Harness-everything/issues/233),
+Phase 3). The `reviewedHoldout` gate requires at least 200 holdout cases, at
+least 50 per language, and `reviewed: true` on every case.
+
+## Gold labels
+
+A gold label answers one question: **which tier should Harness route this prompt
+to, judged from the prompt alone?** The labels follow the tier model in
+[routing.md](routing.md#three-tier-routing-model) and the fixed tier catalog.
+
+| Gold | Use when the prompt asks for | Examples of shape |
+| --- | --- | --- |
+| `tier1` | Something answered or done without changing project behavior: Git/GitHub housekeeping (commit, push, open a PR, sync, tag, post given results as a comment), a status check, reading a log or searching and reporting, the smallest doc fix (a typo, a link, an ignore entry), or a reply the assistant can give directly, including non-engineering chores and discussion questions | "commit these changes", "is issue 88 still open?", "find every repo still using the old logging package", "close the other apps" |
+| `tier2` | A bounded change or investigation: any code, configuration or CI edit (a rename, a value, a revert), running tests or a build, filing an issue, explaining code that must be read first, most bug fixes, a focused review or benchmark, planning or synthesis that produces documents (phase plans, an ADR, an overview), and answers that settle open decisions for work in progress | "rename tmp to buffer", "run npm test", "the CLI fails with this error, fix it", "1. agree 2. keep the old name" |
+| `tier3` | New or changed behavior delivered with tests, and anything larger: a new flag or format, a behavior change across producers and tests, a large refactor, a dependency upgrade with breakage, a migration, a repository-wide change, a multi-phase implementation, or delegation to several agents | "add a --verbose flag with tests", "migrate to ESM", "dispatch agents to audit every repo" |
+| `null` | No actionable content: a bare continuation, "do what you said", a vague reaction, a question about what the assistant meant, or pasted status with no request | "go", "yes", "照剛剛說的改", "something is off here" |
+
+Rules:
+
+1. Label the work the prompt would start. A reply that settles decisions for
+   work in progress ("1. agree 2. use hash") is labeled by that work, not `null`.
+   Use `null` only when no work can be identified at all.
+2. Label the work requested, not the words used. Searching every repository and
+   reporting is `tier1`; changing every repository is `tier3`.
+3. Pick the smallest tier that covers the whole request.
+4. Explicit workflow words ("use fable", "no subagents") do not change the gold.
+   Precedence is a router policy, not a label.
+5. Never derive gold from the lexical router or any model output. A reviewer may
+   read the proposed gold, but the recorded gold is the reviewer's decision.
+
+These definitions are the repository owner's, recorded after the first review
+of the draft: the owner's labels moved many drafted cases one tier up, labeled
+context-dependent replies by their work, and gave non-engineering chores
+`tier1`. The draft's `proposedGold` values predate this and are only
+suggestions. Acknowledgements, reactions, host commands (`/compact`) and meeting
+notes were rejected from the holdout because they never reach routing as tasks.
+
+## Cases
+
+Each draft case has `id`, `family`, `language` (`en` or `zh-TW`), `source`,
+`prompt`, `proposedGold` and a one-line `rationale`.
+
+- **Family.** A family groups prompts that share one task scenario, such as
+  "commit and push" or "fix a CLI error from a pasted log". Paraphrases,
+  translations and variants of one scenario stay in one family, and a family
+  never spans splits. All families in this corpus are `holdout`. Later train or
+  validation data must use new families.
+- **Source.** `synthetic:authored` marks a prompt written for the corpus.
+  `derived:local-history` marks a prompt rewritten from the shape of a real
+  local prompt: same length class, task type and phrasing style, with every
+  name, path, product, customer, issue number and pasted log replaced. No real
+  prompt text, person, organization or local path is committed.
+- **Length.** The corpus follows the observed length profile of real local
+  prompts (median about 90 UTF-8 bytes, about one in five over 224 bytes, a
+  long tail with pasted logs), so that length limits are evaluated on realistic
+  input, not only short prompts.
+
+## Review
+
+A human reviewer decides every case on the review page. The possible decisions are:
+
+| Decision | Effect |
+| --- | --- |
+| `accept` | Gold is the proposed gold. |
+| `relabel` | Gold is the reviewer's chosen label. |
+| `reject` | The case is dropped: ambiguous even for a human, unrealistic, or it leaks private data. |
+
+Each decision records the case ID, the SHA-256 of the exact prompt text it
+reviewed (`promptHash`), the decision, the gold, and the reviewer role. The
+assembler `scripts/system-one-corpus.js` builds the evaluation corpus from the
+draft and the decisions:
+
+- A case with a valid decision becomes `reviewed: true` with the reviewer's gold.
+- A case without a decision, or whose prompt changed after review (hash
+  mismatch), stays `reviewed: false` with the proposed gold, so the gate cannot
+  pass on stale or missing review.
+- A rejected case is omitted.
+- The output passes `validateCorpus`, and the assembler reports per-language and
+  per-gold counts plus whether the `reviewedHoldout` gate would pass.
+
+The agent that drafts cases never records decisions. `reviewed: true` means a
+human reviewed that exact text. Review records store a role
+(`human:repository-owner`), not a name or account. When a decision contradicts
+the owner's decisions on sibling cases, the agent may ask the owner to confirm
+it. The owner's answer is recorded as the decision, and the agent never changes
+a decision on its own.
+
+## Committed files
+
+| File | Content |
+| --- | --- |
+| `benchmarks/fixtures/system-one-holdout-draft.json` | Draft cases with proposed gold |
+| `benchmarks/fixtures/system-one-holdout-reviews.json` | Owner decisions (ID, prompt hash, decision, gold, role) |
+| `benchmarks/fixtures/system-one-holdout.json` | Evaluation corpus, rebuilt with `node scripts/system-one-corpus.js build <draft> <reviews> <corpus>` |
+
+The reviewed holdout has 215 cases (11 rejected): en 93 and zh-TW 122; gold
+`tier1` 47, `tier2` 97, `tier3` 51 and `null` 20. It meets the `reviewedHoldout`
+gate. It says nothing about model quality until a `harness-routing-v1`
+checkpoint is evaluated on it.

package/docs/system-one-routing.md

@@ -41,7 +41,11 @@ never a prompt-supplied executable. No shell or automatic download is used.
 The manifest is version 1 with `python` (executable), `checkpoint` (absolute
 `.safetensors` path), `weightsSha256`, `configSha256` (matching `.json` sidecar),
 `modelId`, `revision`, `domain`, and optional `timeoutMs` (1–10000, default 2000).
-All fields are required except the timeout; unknown fields are rejected. Hashes
+All fields are required except the timeout; unknown fields are rejected. An
+optional `acceptance` object holds exactly `minConfidence` (0.5–0.99) and
+`minMargin` (0–0.9). These are the thresholds calibrated on validation for this
+checkpoint. A scored result carries them, and the router and the evaluator pass
+them to `decide`. Without the field the Phase 0 defaults apply. Hashes
 must be lowercase SHA-256. Revision and domain are artifact-owner declarations,
 not independently verified quality claims. The known `cua-ai/cua-s1-forms` ID
 must use domain `forms-v1`. No pickle format or remote URL is accepted.
@@ -227,7 +231,8 @@ The corpus is `{schemaVersion: 1, cases: [...]}`. Each case has exactly `id`,
 The complete fixed tier catalog must match. IDs and request hashes are unique;
 no family may span splits. Evaluation uses only holdout, never training labels.
 The committed seed corpus is a mechanism fixture, not an independently reviewed
-holdout. Expand and review it separately before making quality claims.
+holdout. The reviewed holdout is written, labeled and assembled as described in
+[system-one-corpus.md](system-one-corpus.md).
 
 The CLI runs the real lexical baseline and the configured provider twice per
 holdout case. Records contain IDs, baseline prediction, decision, model identity,
@@ -278,6 +283,9 @@ unversioned download during a prompt hook.
 | 3 | Corpus and paired evaluator | Leakage checks, coverage/error metrics, repeatability |
 | 4 | Harness training and rollout | Real checkpoint, reviewed holdout, measured host results |
 
+Phase 4 training data, labeling, calibration and release are defined in
+[system-one-training.md](system-one-training.md).
+
 ## Rollout gates
 
 Phase 4 requires a separately reviewed, family-disjoint holdout of at least 200

package/docs/system-one-training.md

@@ -0,0 +1,121 @@
+# System One training data and weights
+
+This document defines how the `harness-routing-v1` tier checkpoint for
+[System One](system-one-routing.md) is trained, calibrated and released
+(issue [#233](https://github.com/dyphn1/Harness-everything/issues/233), Phase 4).
+Gold labels follow [system-one-corpus.md](system-one-corpus.md). The owner-reviewed
+holdout in that document is used only for evaluation. It is never used for training,
+threshold selection or labeler tuning.
+
+## Pipeline
+
+1. **Collect**: copy the owner's own prompts from local assistant histories into a local
+   dataset (`scripts/system-one-collect.js`).
+2. **Label**: an LLM labels each prompt against the owner's rules. The labeler is checked
+   once against the holdout, and the owner spot-checks a sample
+   (`scripts/system-one-label.js`).
+3. **Train**: train a fresh `cua_s1` `tinyx` scorer on the train split and select the epoch
+   on the validation split.
+4. **Calibrate**: choose the confidence and margin thresholds on validation.
+5. **Evaluate**: produce one paired report on the holdout.
+6. **Release**: publish the weights as a GitHub release asset. The installer pins them by
+   SHA-256.
+
+## Sources
+
+Only text the owner typed is collected. Injected context, tool output and assistant text are
+never collected.
+
+| Source | Location | User text |
+| --- | --- | --- |
+| Claude Code | `~/.claude/projects/*/*.jsonl` | `type: "user"` records that are not meta or sidechain, string content or text parts |
+| Codex | `~/.codex/sessions/**/*.jsonl` | `event_msg` `user_message.message`. Older sessions without it fall back to `response_item` user text |
+| Copilot CLI | `~/.copilot/session-state/*/events.jsonl` | `user.message` `data.content` |
+| VS Code Copilot Chat | `%APPDATA%/Code/User/workspaceStorage/*/chatSessions/*.json(l)` and `globalStorage/emptyWindowChatSessions` | `requests[].message.text` in snapshots (`kind: 0`) and request pushes (`kind: 2`, key `["requests"]`) |
+
+The following are dropped:
+
+- Text that starts with an injected wrapper tag, such as `<environment_context>`,
+  `<system-reminder>`, `<command-…>`, `<task-notification>` or `<ide_…>`.
+- Tool results.
+- Slash commands.
+- Session-continuation summaries ("This session is being continued…").
+- Empty text.
+
+Codex sessions include prompts that another agent dispatched. They are kept and tagged with
+their source, because Harness routes them in Codex too.
+
+## Filters, families and splits
+
+- **Length**: at most 1024 UTF-8 bytes, which matches the checkpoint's `context_tokens`. A
+  longer prompt is still refused at inference and falls back to lexical routing. It is never
+  truncated.
+- **Deduplication**: exact duplicates after whitespace normalization are dropped. The first
+  occurrence is kept.
+- **Holdout leakage**: a prompt is dropped if its character-trigram Jaccard similarity to any
+  holdout prompt is at least 0.5. The same applies if either prompt contains the other after
+  normalization and the shorter one is more than 12 characters long. This matters because
+  the holdout includes rewrites of local prompt shapes.
+- **Family**: the source session. Prompts from one conversation never span splits.
+- **Split**: 15% of families go to `validation`, chosen by a stable SHA-256 of the family ID.
+  The rest go to `train`.
+
+## Privacy
+
+- Collected prompts and labels stay on the local machine under
+  `~/.agents/harness-everything/system-one/training/` and are **never committed**. The
+  repository holds only the scripts, aggregate statistics and the SHA-256 of each dataset
+  file.
+- Email addresses and user home paths are replaced with placeholders before storage. Names
+  and product terms are not reliably detectable and remain.
+- The released weights come from a 4-way byte-level classifier with about 0.7M parameters. It
+  has no text-generation head, but a trained model can still encode fragments of its data.
+  The owner accepted this for their own prompts.
+- Labeling sends the prompts to the Claude API. The owner authorized this.
+
+## Labeling
+
+- **Labeler**: `claude -p` with Sonnet.
+  - Flags: `--setting-sources ""` (no hooks or plugins), `--no-session-persistence` (labeling
+    sessions never become collectable history), `--tools ""` and a JSON schema.
+  - The instructions contain the gold table and rules from
+    [system-one-corpus.md](system-one-corpus.md).
+  - Prompts go in batches of 100. Every returned index must be labeled exactly once with
+    `tier1`, `tier2`, `tier3` or `null`. A malformed batch is retried once and then
+    recorded as failed. It is never partially accepted.
+- **Labeler check**: the labeler labels the 215 holdout prompts once, before any training
+  data is labeled. The report gives agreement with the owner's gold and the per-class recall.
+  - Labeling proceeds only when agreement is at least 80%.
+  - The instructions are never tuned on the holdout. If the check fails, the owner reviews
+    the disagreements and decides how to proceed.
+- **Owner spot-check**: a random 5% sample of the labeled training data is reviewed on the
+  review page, and the disagreement rate is reported. The owner's decisions replace the LLM
+  labels for those cases.
+
+## Training and calibration
+
+- **Model**: a new `cua_s1` `tinyx` scorer built with `make_system`.
+  - Config: `width` 128, `rank` 128, `layers` 2, `heads` 4, `context_tokens` 1024 and
+    `option_tokens` 96.
+  - Weights are initialized fresh, not from the forms checkpoint, whose domain and
+    positional range differ.
+- **Target**: each example is the fixed tier catalog with the index of the gold option.
+  `null` maps to `unclassified`.
+- **Optimization**:
+  - Cross-entropy, AdamW, a fixed seed and CPU.
+  - The epoch with the lowest validation negative log-likelihood is kept.
+  - The checkpoint metadata records the dataset hashes, the config and the validation metrics.
+- **Calibration**: `minConfidence` and `minMargin` are chosen on validation to maximize
+  coverage, subject to an accepted precision of at least 98%. The router currently uses the
+  Phase 0 defaults (0.9 and 0.2). Carrying calibrated thresholds into the manifest is a
+  separate contract change.
+
+## Evaluation and release
+
+- **Evaluation**: `scripts/evaluate-system-one.js` runs once on
+  `benchmarks/fixtures/system-one-holdout.json` with the new manifest. The rollout gates and
+  the lexical baseline (macro-F1 0.254) decide the result. The report goes under
+  `benchmarks/results/system-one/`.
+- **Release**: the `.safetensors` and `.json` files are attached to a GitHub release. The
+  installer's pins gain the new model ID, revision, `domain: "harness-routing-v1"` and both
+  SHA-256 values. Until every rollout gate passes, the router default stays `off`.

package/harness-everything/SKILL.md

@@ -4,7 +4,7 @@ description: "Route software/project work through Harness: classify tier/topolog
 license: Apache-2.0
 metadata:
   author: Miya Daniel
-  version: 0.22.1
+  version: 0.24.0
 ---
 
 # Harness Everything

package/harness-everything/scripts/system-one/provider.js

@@ -10,10 +10,17 @@ const unavailable = reason => ({ status: 'unavailable', reason });
 const exact = (value, keys) => value && typeof value === 'object' && !Array.isArray(value)
   && Object.keys(value).length === keys.length && keys.every(k => Object.hasOwn(value, k));
 const label = value => typeof value === 'string' && value.length > 0 && value.length <= 128 && !/[\u0000-\u001f]/.test(value);
+// Thresholds calibrated on validation for this checkpoint (docs/system-one-training.md).
+const inRange = (v, lo, hi) => typeof v === 'number' && Number.isFinite(v) && v >= lo && v <= hi;
+const validAcceptance = a => exact(a, ['minConfidence', 'minMargin']) && inRange(a.minConfidence, 0.5, 0.99) && inRange(a.minMargin, 0, 0.9);
+// A scored result carries the manifest's thresholds so every caller decides with them.
+const withAcceptance = (result, manifest) => (result && result.status === 'scored' && manifest.acceptance
+  ? { ...result, acceptance: { ...manifest.acceptance } } : result);
 function validateManifest(m) {
   const required = ['schemaVersion', 'python', 'checkpoint', 'weightsSha256', 'configSha256', 'modelId', 'revision', 'domain'];
   if (!m || typeof m !== 'object' || Array.isArray(m) || required.some(k => !Object.hasOwn(m, k))
-    || Object.keys(m).some(k => ![...required, 'timeoutMs', 'transport', 'idleTimeoutMs'].includes(k)) || m.schemaVersion !== 1
+    || Object.keys(m).some(k => ![...required, 'timeoutMs', 'transport', 'idleTimeoutMs', 'acceptance'].includes(k)) || m.schemaVersion !== 1
+    || (m.acceptance !== undefined && !validAcceptance(m.acceptance))
     || ['python', 'modelId', 'revision', 'domain'].some(k => typeof m[k] !== 'string' || !m[k].trim() || m[k].length > 4096 || m[k].includes('\0'))
     || typeof m.checkpoint !== 'string' || !path.isAbsolute(m.checkpoint) || path.extname(m.checkpoint) !== '.safetensors'
     || ['weightsSha256', 'configSha256'].some(k => typeof m[k] !== 'string' || !/^[a-f0-9]{64}$/.test(m[k]))
@@ -50,8 +57,8 @@ function score(request, manifestPath) {
   validateRequest(request);
   let manifest;
   try { manifest = readManifest(manifestPath); } catch (_) { return unavailable('provider-config'); }
-  if (manifest.transport === 'resident') return require('./resident').score({ ...request }, manifest, manifestPath);
-  return oneShot(request, manifest);
+  if (manifest.transport === 'resident') return withAcceptance(require('./resident').score({ ...request }, manifest, manifestPath), manifest);
+  return withAcceptance(oneShot(request, manifest), manifest);
 }
 // Async callers (the hook entry) reach a resident server without the sync worker bridge.
 // One-shot has no server to overlap with, so it runs the same child synchronously.
@@ -59,8 +66,8 @@ async function scoreAsync(request, manifestPath) {
   validateRequest(request);
   let manifest;
   try { manifest = readManifest(manifestPath); } catch (_) { return unavailable('provider-config'); }
-  if (manifest.transport === 'resident') return require('./resident').scoreAsync({ ...request }, manifest, manifestPath);
-  return oneShot(request, manifest);
+  if (manifest.transport === 'resident') return withAcceptance(await require('./resident').scoreAsync({ ...request }, manifest, manifestPath), manifest);
+  return withAcceptance(oneShot(request, manifest), manifest);
 }
 // Send the exact validated manifest to the child on stdin: no config reread race.
 function oneShot(request, manifest) {

package/harness-everything/scripts/system-one/router.js

@@ -26,7 +26,7 @@ function selectTier(input, env = process.env, scorer = provider.score) {
     const allowed = ['provider-timeout', 'provider-output-limit', 'provider-exit', 'provider-json', 'provider-config', 'provider-unavailable', 'provider-starting'];
     return fallback(allowed.includes(result?.reason) ? result.reason : 'provider-unavailable');
   }
-  const decision = decide(request, result.response);
+  const decision = decide(request, result.response, result.acceptance ? { ...result.acceptance } : {});
   if (decision.selectedId === 'unclassified') Object.assign(decision, { selectedId: null, status: 'abstain', reason: 'unclassified' });
   const accepted = decision.status === 'accepted';
   const belowFloor = accepted && input.floor != null && RANK[decision.selectedId] < RANK[input.floor];

package/opencode-plugin/plugin.json

@@ -1,7 +1,7 @@
 {
   "name": "harness-enforcement",
   "description": "Repository inventory for the opencode enforcement plugin and its standalone helper",
-  "version": "0.22.1",
+  "version": "0.24.0",
   "runtime": "index.mjs",
   "hooks": {
     "verify": "hooks/verify.js"

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "harness-everything",
-  "version": "0.22.1",
+  "version": "0.24.0",
   "description": "Harness OS: An Orchestrated Agent Operating System for AI-driven Development",
   "main": "harness-everything/scripts/bootstrap.js",
   "bin": {

package/plugins/harness-everything/.codex-plugin/plugin.json

@@ -1,6 +1,6 @@
 {
   "name": "harness-everything",
-  "version": "0.22.1",
+  "version": "0.24.0",
   "description": "Invariant-first engineering harness for routing, verification, and failure recovery while preserving agent autonomy.",
   "author": {
     "name": "Miya Daniel",

package/plugins/harness-everything/plugin.json

@@ -1,7 +1,7 @@
 {
   "$schema": "https://agent-plugins.org/schemas/1.0.0/plugin.schema.json",
   "name": "harness-everything",
-  "version": "0.22.1",
+  "version": "0.24.0",
   "description": "Invariant-first engineering harness for routing, verification, and failure recovery while preserving agent autonomy.",
   "author": {
     "name": "Miya Daniel",

package/plugins/harness-everything/skills/harness-everything/SKILL.md

@@ -4,7 +4,7 @@ description: "Route software/project work through Harness: classify tier/topolog
 license: Apache-2.0
 metadata:
   author: Miya Daniel
-  version: 0.22.1
+  version: 0.24.0
 ---
 
 # Harness Everything

package/plugins/harness-everything/skills/harness-everything/scripts/system-one/provider.js

@@ -10,10 +10,17 @@ const unavailable = reason => ({ status: 'unavailable', reason });
 const exact = (value, keys) => value && typeof value === 'object' && !Array.isArray(value)
   && Object.keys(value).length === keys.length && keys.every(k => Object.hasOwn(value, k));
 const label = value => typeof value === 'string' && value.length > 0 && value.length <= 128 && !/[\u0000-\u001f]/.test(value);
+// Thresholds calibrated on validation for this checkpoint (docs/system-one-training.md).
+const inRange = (v, lo, hi) => typeof v === 'number' && Number.isFinite(v) && v >= lo && v <= hi;
+const validAcceptance = a => exact(a, ['minConfidence', 'minMargin']) && inRange(a.minConfidence, 0.5, 0.99) && inRange(a.minMargin, 0, 0.9);
+// A scored result carries the manifest's thresholds so every caller decides with them.
+const withAcceptance = (result, manifest) => (result && result.status === 'scored' && manifest.acceptance
+  ? { ...result, acceptance: { ...manifest.acceptance } } : result);
 function validateManifest(m) {
   const required = ['schemaVersion', 'python', 'checkpoint', 'weightsSha256', 'configSha256', 'modelId', 'revision', 'domain'];
   if (!m || typeof m !== 'object' || Array.isArray(m) || required.some(k => !Object.hasOwn(m, k))
-    || Object.keys(m).some(k => ![...required, 'timeoutMs', 'transport', 'idleTimeoutMs'].includes(k)) || m.schemaVersion !== 1
+    || Object.keys(m).some(k => ![...required, 'timeoutMs', 'transport', 'idleTimeoutMs', 'acceptance'].includes(k)) || m.schemaVersion !== 1
+    || (m.acceptance !== undefined && !validAcceptance(m.acceptance))
     || ['python', 'modelId', 'revision', 'domain'].some(k => typeof m[k] !== 'string' || !m[k].trim() || m[k].length > 4096 || m[k].includes('\0'))
     || typeof m.checkpoint !== 'string' || !path.isAbsolute(m.checkpoint) || path.extname(m.checkpoint) !== '.safetensors'
     || ['weightsSha256', 'configSha256'].some(k => typeof m[k] !== 'string' || !/^[a-f0-9]{64}$/.test(m[k]))
@@ -50,8 +57,8 @@ function score(request, manifestPath) {
   validateRequest(request);
   let manifest;
   try { manifest = readManifest(manifestPath); } catch (_) { return unavailable('provider-config'); }
-  if (manifest.transport === 'resident') return require('./resident').score({ ...request }, manifest, manifestPath);
-  return oneShot(request, manifest);
+  if (manifest.transport === 'resident') return withAcceptance(require('./resident').score({ ...request }, manifest, manifestPath), manifest);
+  return withAcceptance(oneShot(request, manifest), manifest);
 }
 // Async callers (the hook entry) reach a resident server without the sync worker bridge.
 // One-shot has no server to overlap with, so it runs the same child synchronously.
@@ -59,8 +66,8 @@ async function scoreAsync(request, manifestPath) {
   validateRequest(request);
   let manifest;
   try { manifest = readManifest(manifestPath); } catch (_) { return unavailable('provider-config'); }
-  if (manifest.transport === 'resident') return require('./resident').scoreAsync({ ...request }, manifest, manifestPath);
-  return oneShot(request, manifest);
+  if (manifest.transport === 'resident') return withAcceptance(await require('./resident').scoreAsync({ ...request }, manifest, manifestPath), manifest);
+  return withAcceptance(oneShot(request, manifest), manifest);
 }
 // Send the exact validated manifest to the child on stdin: no config reread race.
 function oneShot(request, manifest) {

package/plugins/harness-everything/skills/harness-everything/scripts/system-one/router.js

@@ -26,7 +26,7 @@ function selectTier(input, env = process.env, scorer = provider.score) {
     const allowed = ['provider-timeout', 'provider-output-limit', 'provider-exit', 'provider-json', 'provider-config', 'provider-unavailable', 'provider-starting'];
     return fallback(allowed.includes(result?.reason) ? result.reason : 'provider-unavailable');
   }
-  const decision = decide(request, result.response);
+  const decision = decide(request, result.response, result.acceptance ? { ...result.acceptance } : {});
   if (decision.selectedId === 'unclassified') Object.assign(decision, { selectedId: null, status: 'abstain', reason: 'unclassified' });
   const accepted = decision.status === 'accepted';
   const belowFloor = accepted && input.floor != null && RANK[decision.selectedId] < RANK[input.floor];

package/sbom.cdx.json

@@ -2,23 +2,23 @@
   "$schema": "https://cyclonedx.org/schema/bom-1.7.schema.json",
   "bomFormat": "CycloneDX",
   "specVersion": "1.7",
-  "serialNumber": "urn:uuid:3cb9e1e8-a9f0-5bc4-87ab-794acb1e79c3",
+  "serialNumber": "urn:uuid:5f8c3ad1-23b9-55c5-a389-ff6c3877f9dd",
   "version": 1,
   "metadata": {
-    "timestamp": "2026-09-23T11:35:56.699Z",
+    "timestamp": "2026-09-23T13:40:29.695Z",
     "tools": [
       {
         "vendor": "Harness Everything",
         "name": "generate-sbom.js",
-        "version": "0.22.1"
+        "version": "0.24.0"
       }
     ],
     "component": {
       "type": "application",
-      "bom-ref": "pkg:npm/harness-everything@0.22.1",
+      "bom-ref": "pkg:npm/harness-everything@0.24.0",
       "name": "harness-everything",
-      "version": "0.22.1",
-      "purl": "pkg:npm/harness-everything@0.22.1"
+      "version": "0.24.0",
+      "purl": "pkg:npm/harness-everything@0.24.0"
     }
   },
   "components": [
@@ -1999,7 +1999,7 @@
       ]
     },
     {
-      "ref": "pkg:npm/harness-everything@0.22.1",
+      "ref": "pkg:npm/harness-everything@0.24.0",
       "dependsOn": [
         "pkg:npm/%40commitlint/cli@19.8.1",
         "pkg:npm/%40commitlint/config-conventional@19.8.1",

package/scripts/evaluate-system-one.js

@@ -29,7 +29,7 @@ async function run(corpus, manifest) {
       for (let i = 0; i < 2; i++) {
         const start = performance.now();
         const result = await scoreAsync(c.request, manifest);
-        const decision = result.status === 'scored' ? decide(c.request, result.response) : result;
+        const decision = result.status === 'scored' ? decide(c.request, result.response, result.acceptance ? { ...result.acceptance } : {}) : result;
         if (decision.selectedId === 'unclassified') Object.assign(decision, { status: 'abstain', reason: 'unclassified', selectedId: null });
         const scores = ['accepted', 'abstain'].includes(decision.status) ? result.response.scores.map(s => s.probability) : null;
         runs.push({ decision, scores, latencyMs: performance.now() - start, coldStart: !warm });

package/scripts/system-one-calibrate.js

@@ -0,0 +1,44 @@
+#!/usr/bin/env node
+'use strict';
+// Chooses System One acceptance thresholds on validation scores (docs/system-one-training.md).
+// The holdout is never used here.
+const fs = require('node:fs');
+
+const IDS = ['tier1', 'tier2', 'tier3', 'unclassified'];
+// Same acceptance rule as contract.decide: top probability and top-minus-second margin, unclassified abstains.
+function decideRow(probs, { minConfidence, minMargin }) {
+  const order = probs.map((p, i) => [p, i]).sort((a, b) => b[0] - a[0] || a[1] - b[1]);
+  const [top, second] = order;
+  const margin = top[0] - second[0];
+  if (margin === 0 || top[0] < minConfidence || margin < minMargin || IDS[top[1]] === 'unclassified') return null;
+  return IDS[top[1]];
+}
+function calibrate(rows, { minPrecision = 0.98 } = {}) {
+  let best = null;
+  for (let c = 50; c <= 99; c++) {
+    for (let m = 0; m <= 90; m += 5) {
+      const t = { minConfidence: c / 100, minMargin: m / 100 };
+      let accepted = 0; let correct = 0;
+      for (const r of rows) { const d = decideRow(r.probs, t); if (d) { accepted++; if (d === (r.gold === null ? 'unclassified' : r.gold)) correct++; } }
+      const precision = accepted ? correct / accepted : 0;
+      if (!accepted || precision < minPrecision) continue;
+      const coverage = accepted / rows.length;
+      // Highest coverage wins; ties keep the first (lowest) thresholds scanned.
+      if (!best || coverage > best.coverage) best = { ...t, coverage, precision, accepted };
+    }
+  }
+  return best ? { feasible: true, rows: rows.length, minPrecision, ...best }
+    : { feasible: false, rows: rows.length, minPrecision, minConfidence: null, minMargin: null, coverage: 0, precision: null, accepted: 0 };
+}
+
+if (require.main === module) {
+  try {
+    const [input, output, extra] = process.argv.slice(2);
+    if (!input || !output || extra) throw new Error('usage: system-one-calibrate.js <val-scores.jsonl> <calibration.json>');
+    const rows = fs.readFileSync(input, 'utf8').split('\n').filter(Boolean).map(l => JSON.parse(l));
+    const result = calibrate(rows);
+    fs.writeFileSync(output, `${JSON.stringify(result, null, 2)}\n`);
+    console.log(JSON.stringify(result));
+  } catch (err) { console.error(`system-one-calibrate: ${err.message}`); process.exitCode = 1; }
+}
+module.exports = { decideRow, calibrate };