harness-everything 0.20.7 → 0.21.0

package/docs/system-one-routing.md

@@ -0,0 +1,202 @@
+# System One routing migration
+
+Status: experimental engineering; issue [#233](https://github.com/dyphn1/Harness-everything/issues/233).
+The default router stays lexical until a Harness checkpoint passes the rollout gates.
+Implement each phase in **documentation → failing tests → implementation** order.
+
+## Phase 0: scorer contract
+
+System One scores a complete, caller-owned candidate list in one pass. It cannot
+generate a workflow, execute actions, authorize tools, or write memory. Policy and
+contract assembly remain deterministic. This boundary does not add execution locks.
+
+`createRequest(task, context, options)` returns a version-1 request. Task names and
+candidate IDs use lowercase letters, digits, dots, underscores or hyphens (1–80
+characters). Context is nonempty UTF-8 (at most 65,536 bytes). There are 2–256
+unique `{id, text}` options; text is nonempty and at most 4,096 bytes. Empty,
+malformed, unknown-field and wrong-type inputs are errors. Input objects are not
+mutated. Candidate order matters. `catalogHash` is SHA-256 of JSON-encoded
+`{task, options}`; `requestHash` additionally binds the exact context. Neither
+fingerprint grants permission or authenticates a model.
+
+The provider response has exactly `schemaVersion`, `requestHash`, `catalogHash`,
+`model`, and `scores`. Model has nonempty `id`, `revision`, and `domain` strings.
+Scores have exactly `{id, probability}` in request order. All candidates occur
+exactly once; probabilities are finite numbers in [0, 1], summing to one within
+1e-6. Unknown fields, stale request/catalog hashes, and version mismatches fail.
+
+`decide(request, response, policy)` returns exactly `status`, `reason`,
+`selectedId`, `confidence`, `margin`, and `model`. Status is `accepted`, `abstain`,
+or `invalid-output`; nonaccepted results always have `selectedId: null`.
+Policy requires a domain (default `harness-routing-v1`), minimum confidence
+(default 0.9) and minimum margin (default 0.2). Invalid policy is a caller error.
+Reject domain mismatch, tied winners, low confidence and low margin. A softmax
+score is not calibrated correctness probability. Invalid output returns no raw
+response, exception, or prompt. Identical inputs give identical decisions.
+
+## Phase 1: local provider
+
+`provider.score(request, manifestPath)` uses a trusted host-configured manifest,
+never a prompt-supplied executable. No shell or automatic download is used.
+The manifest is version 1 with `python` (executable), `checkpoint` (absolute
+`.safetensors` path), `weightsSha256`, `configSha256` (matching `.json` sidecar),
+`modelId`, `revision`, `domain`, and optional `timeoutMs` (1–10000, default 2000).
+All fields are required except the timeout; unknown fields are rejected. Hashes
+must be lowercase SHA-256. Revision and domain are artifact-owner declarations,
+not independently verified quality claims. The known `cua-ai/cua-s1-forms` ID
+must use domain `forms-v1`. No pickle format or remote URL is accepted.
+
+Install the optional `cua_s1` source in a dedicated Python 3.11–3.13 environment
+from upstream revision `b7f7e2d8714609853a29c7d049140bc46aec0954`, using the
+`libs/cua-s1/python` project. This npm package does not install Python, torch or
+weights. Configure its Python executable in the manifest. A manifest alone does
+not prove which Python package was installed, so `provider.provenance(manifestPath)`
+runs `cua_adapter.py --provenance` with the configured interpreter. The probe
+reads installed metadata only (no torch import, no checkpoint load) and reports
+the Python implementation/version, the distribution providing `cua_s1`, its
+version, the torch version, and the source revision. Only a PEP 610
+`direct_url.json` VCS `commit_id` counts as a source revision, so install from a
+VCS URL pinned to that commit (not an editable or local-directory install).
+The status is `pinned` when it matches, `mismatch` otherwise, or `unavailable`
+when no revision is recorded. Malformed probe output is `provider-json`.
+
+`npm run system-one:install` (or `node harness-everything/scripts/system-one/install.js`)
+performs these steps on explicit request; hooks never run it. It selects a
+Python 3.11–3.13 interpreter (`--python` overrides; 3.14 is rejected by upstream),
+creates `<dir>/venv`, installs CPU-only torch from the PyTorch CPU index, then
+installs the pinned `cua_s1` Git URL. It downloads the `cua-ai/cua-s1-forms`
+safetensors weights and JSON sidecar at Hugging Face revision
+`f54adbf447f4ca6ec259f529ee3f2e3e09f8cc71`, verifies pinned sizes and SHA-256
+before an atomic write (the pickle `.pt` is never fetched), and writes
+`<dir>/manifest.json` with `timeoutMs: 10000`. The default `<dir>` is
+`~/.agents/harness-everything/system-one`; `--dir` takes an absolute path and
+`--dry-run` prints the plan without changes. Acceptance requires a `pinned`
+provenance probe and one real CPU inference; otherwise the exit code is 1.
+Torch/numpy/safetensors versions resolve at install time and are recorded by
+the probe. The installer does not change the router mode: it prints
+`HARNESS_SYSTEM_ONE_MODE=shadow` and the manifest path for the host
+environment. This checkpoint declares domain `forms-v1`, so shadow diagnostics
+report `domain-mismatch` and prefer never applies it. It also rejects context
+above its 224-byte limit (`provider-exit`), so installing it proves the local
+inference path only, not Harness routing quality.
+
+The adapter verifies both files before `load_checkpoint(..., 'cpu')`, sets one
+torch thread and deterministic inference, calls the upstream collator/model,
+and returns a probability for every option. It rejects context or option text
+that exceeds the loaded byte limits before collation (no silent truncation).
+No prompts or model values are written to disk or echoed in error messages.
+
+The Node transport has a hard timeout and a 1 MiB stdout/stderr bound; it kills
+the direct child on timeout. It starts one short-lived Python process per score
+request. This is an evaluation bridge, not a warm, persistent low-latency server.
+The process cannot execute actions and is not given a CUA driver. Failures return
+`{status: 'unavailable', reason}` with a bounded reason code; malformed score
+responses are checked by Phase 0. Transport fixtures prove IPC and failure
+handling only. Real checkpoint inference and measured CPU latency remain a
+separate gate, with explicit unavailable evidence if dependencies are absent.
+
+## Phase 2: tier integration and rollback
+
+Set `HARNESS_SYSTEM_ONE_MODE=off|shadow|prefer` in the host environment. Default
+is `off`: no provider process, diagnostic, or contract change. Supply an absolute
+`HARNESS_SYSTEM_ONE_CONFIG` path to the Phase 1 manifest for the other modes.
+An unknown mode reports `invalid-mode` and leaves the lexical result unchanged.
+Shadow records the decision but never changes the route. Prefer replaces only
+the recommended tier when a valid `harness-routing-v1` result passes Phase 0
+thresholds. The full fixed tier catalog (tier1, tier2, tier3, unclassified) is
+always scored; it is never filtered using lexical matches. `unclassified` means
+abstention and keeps the existing route. Explicit strategy/model requests keep
+precedence and prevent model tier replacement. Set mode back to `off` to roll back.
+
+Prefer can raise a tier but never lower it below the deterministic structural
+floor: a macro-scope signal sets floor `tier3`, and multi-task, multi-sentence
+structure (other than a trivial docs edit) sets floor `tier2`. An accepted result
+below the floor is not applied and reports `structural-floor`, so repository-wide
+work keeps its Tier 3 strategy, independent verification and worktree
+invariants. Tier-3 keywords are lexical guesses, not structural floors. The
+policy assembler still consumes the original structural and risk signals.
+Safety, explicit prohibitions, memory ownership, and verification are not model
+outputs. This phase does not yet replace guide matching, dynamic
+skill triggers, fact-audit reminders, or the structural workflow signals.
+The same scorer API can evaluate caller-supplied text options; additional routing
+surfaces require their own catalog and held-out evidence before integration.
+
+When enabled, a single `SYSTEM ONE` JSON diagnostic reports mode, status, reason,
+selected candidate, model identity, confidence and margin; it omits prompt text,
+score vectors and artifact paths. The existing router contract schema is unchanged.
+Kernel/packaged entry points inherit these host environment variables. Prompts
+and hook payload fields cannot set model configuration or launch executables.
+Repository tests establish mechanism/package evidence only; actual host loading
+and semantic quality remain unverified until retained sessions demonstrate them.
+
+## Phase 3: corpus and paired evaluation
+
+Use `node scripts/evaluate-system-one.js corpus.json /absolute/manifest.json report.json`.
+The corpus is `{schemaVersion: 1, cases: [...]}`. Each case has exactly `id`,
+`family`, `split` (`train|validation|holdout`), `language` (`en|zh-TW`), `source`
+(human-reviewed provenance, not baseline-generated gold), `reviewed` (boolean),
+`request` (Phase 0 tier request), and `gold` (candidate ID, or null for abstention).
+The complete fixed tier catalog must match. IDs and request hashes are unique;
+no family may span splits. Evaluation uses only holdout, never training labels.
+The committed seed corpus is a mechanism fixture, not an independently reviewed
+holdout. Expand and review it separately before making quality claims.
+
+The CLI runs the real lexical baseline and the configured provider twice per
+holdout case. Records contain IDs, baseline prediction, decision, model identity,
+the catalog-order probability vector (`scores`; null for unavailable or invalid
+output), latency, and `coldStart: true`; they do not copy prompts into the
+report. A scored run's confidence and margin must match its vector. Gold
+`unclassified` is represented by null. Nonaccepted model results also predict
+null. The evaluator rejects missing, duplicate or unknown cases, unsupported
+predictions, malformed measurements, and inconsistent model identities. Reruns
+compare the full decision, model identity and score vector, but not measured
+latency: equal top-2 values with a reordered tail still disagree.
+No normalization is permitted for candidate IDs or model revision.
+
+Report both baseline and model accuracy/macro-F1, model coverage, accepted
+precision/error rate, abstention, repeated-decision agreement, and separate
+cold/warm p95. Zero accepted decisions has null precision and never passes.
+Unavailable responses remain abstentions and count against coverage. Null gold
+is included in macro-F1's class set; the four catalog labels are always included
+even when unobserved, preventing favorable per-run class selection.
+
+The evaluator emits explicit rollout checks, not an automatic deployment. It
+requires reviewed holdout counts, semantic/coverage thresholds, repeatability,
+warm measurements, a `pinned` source revision, plus independently retained
+policy and live-host evidence. The report's `evidence.source` keeps the probe
+result (including `mismatch` or `unavailable`) and `environment.python`.
+The latter evidence is not supplied by this offline runner, so it always reports
+those gates pending. With the one-shot adapter, all latency is cold and the warm
+gate also remains pending. Exporting a report never changes router defaults.
+
+## Model suitability
+
+The [CUA-S1-FORMS model card](https://huggingface.co/cua-ai/cua-s1-forms)
+describes a form-specific option scorer, with English-centric training and short
+byte contexts. These results do not establish Harness routing quality. Its
+checkpoint is an experimental baseline, not a production routing checkpoint.
+The [upstream source](https://github.com/trycua/cua/tree/main/libs/cua-s1)
+has a source-only release description; source and checkpoint provenance/licenses
+must be checked independently. Do not infer a published Python wheel or use an
+unversioned download during a prompt hook.
+
+## Planned phases
+
+| Phase | Deliverable | Acceptance evidence |
+| --- | --- | --- |
+| 0 | Versioned option-scoring boundary | Complete positive/negative contract tests |
+| 1 | Local CUA-S1 adapter | Bounded subprocess, artifact validation, explicit failures |
+| 2 | Router integration | Off/shadow/prefer, existing policy invariants, rollback |
+| 3 | Corpus and paired evaluator | Leakage checks, coverage/error metrics, repeatability |
+| 4 | Harness training and rollout | Real checkpoint, reviewed holdout, measured host results |
+
+## Rollout gates
+
+Phase 4 requires a separately reviewed, family-disjoint holdout of at least 200
+cases, at least 50 each in English and Traditional Chinese. Accepted precision
+must be ≥98%, coverage ≥80%, and macro-F1 at least the lexical baseline.
+Policy invariants must remain intact in every case; repeat decisions twice with
+100% agreement. Record checkpoint/source hashes, CPU, OS, runtime, thread count,
+cold latency and warm p95 (target ≤100ms). Synthetic fixtures and mock providers
+only validate mechanisms. They cannot satisfy these model/behavior gates.
+Until measured evidence passes, retain the default, fallback and this open issue.

package/harness-everything/SKILL.md

@@ -4,7 +4,7 @@ description: "Route software/project work through Harness: classify tier/topolog
 license: Apache-2.0
 metadata:
   author: Miya Daniel
-  version: 0.19.1
+  version: 0.21.0
 ---
 
 # Harness Everything

package/harness-everything/scripts/system-one/contract.js

@@ -0,0 +1,45 @@
+'use strict';
+const { createHash } = require('node:crypto');
+const hash = value => createHash('sha256').update(JSON.stringify(value)).digest('hex');
+const exact = (value, keys) => value && typeof value === 'object' && !Array.isArray(value)
+  && Object.keys(value).length === keys.length && keys.every(k => Object.hasOwn(value, k));
+const text = (value, max = 4096) => typeof value === 'string' && value.trim().length > 0 && Buffer.byteLength(value, 'utf8') <= max;
+const id = value => typeof value === 'string' && /^[a-z0-9._-]{1,80}$/.test(value);
+const probability = value => typeof value === 'number' && Number.isFinite(value) && value >= 0 && value <= 1;
+function fail(code) { throw Object.assign(new TypeError(code), { code }); }
+function createRequest(task, context, options) {
+  if (!id(task) || !text(context, 65536) || !Array.isArray(options) || options.length < 2 || options.length > 256
+    || options.some(o => !exact(o, ['id', 'text']) || !id(o.id) || !text(o.text))
+    || new Set(options.map(o => o.id)).size !== options.length) fail('invalid-request');
+  const copy = options.map(o => ({ id: o.id, text: o.text }));
+  const catalogHash = hash({ task, options: copy });
+  return { schemaVersion: 1, task, context, options: copy, catalogHash, requestHash: hash({ catalogHash, context }) };
+}
+function validateRequest(req) {
+  if (!exact(req, ['schemaVersion', 'task', 'context', 'options', 'catalogHash', 'requestHash']) || req.schemaVersion !== 1) fail('invalid-request');
+  const rebuilt = createRequest(req.task, req.context, req.options);
+  if (req.catalogHash !== rebuilt.catalogHash || req.requestHash !== rebuilt.requestHash) fail('invalid-request');
+  return true;
+}
+function decide(req, res, policy = {}) {
+  validateRequest(req);
+  if (!policy || typeof policy !== 'object' || Array.isArray(policy)
+    || Object.keys(policy).some(k => !['domain', 'minConfidence', 'minMargin'].includes(k))) fail('invalid-policy');
+  const { domain = 'harness-routing-v1', minConfidence = 0.9, minMargin = 0.2 } = policy;
+  if (!text(domain) || !probability(minConfidence) || !probability(minMargin)) fail('invalid-policy');
+  const empty = { status: 'invalid-output', reason: 'invalid-response', selectedId: null, confidence: null, margin: null, model: null };
+  if (!exact(res, ['schemaVersion', 'requestHash', 'catalogHash', 'model', 'scores'])
+    || res.schemaVersion !== 1 || res.requestHash !== req.requestHash || res.catalogHash !== req.catalogHash
+    || !exact(res.model, ['id', 'revision', 'domain']) || !Object.values(res.model).every(v => text(v))
+    || !Array.isArray(res.scores) || res.scores.length !== req.options.length
+    || res.scores.some((s, i) => !exact(s, ['id', 'probability']) || s.id !== req.options[i].id || !probability(s.probability))
+    || Math.abs(res.scores.reduce((s, o) => s + o.probability, 0) - 1) > 1e-6) return empty;
+  const ranked = [...res.scores].sort((a, b) => b.probability - a.probability);
+  const confidence = ranked[0].probability;
+  const margin = confidence - ranked[1].probability;
+  const reason = res.model.domain !== domain ? 'domain-mismatch' : margin === 0 ? 'tie'
+    : confidence < minConfidence ? 'low-confidence' : margin < minMargin ? 'low-margin' : 'scored';
+  return { status: reason === 'scored' ? 'accepted' : 'abstain', reason,
+    selectedId: reason === 'scored' ? ranked[0].id : null, confidence, margin, model: { ...res.model } };
+}
+module.exports = { createRequest, validateRequest, decide };

package/harness-everything/scripts/system-one/cua_adapter.py

@@ -0,0 +1,94 @@
+"""Optional CPU CUA-S1 bridge. Stdout is protocol-only; never execute UI actions."""
+import hashlib
+import json
+from pathlib import Path
+import re
+import sys
+
+
+def verify_artifacts(manifest):
+    weights = Path(manifest['checkpoint'])
+    if not weights.is_absolute() or weights.suffix != '.safetensors':
+        raise ValueError('checkpoint-format')
+    for file, key in [(weights, 'weightsSha256'), (weights.with_suffix('.json'), 'configSha256')]:
+        with file.open('rb') as handle:
+            digest = hashlib.file_digest(handle, 'sha256').hexdigest()
+        if digest != manifest[key]:
+            raise ValueError('artifact-hash')
+    return weights
+
+
+def check_limits(request, config):
+    if len(request['context'].encode('utf-8')) > config['context_tokens'] or any(
+        len(option['text'].encode('utf-8')) > config['option_tokens'] for option in request['options']
+    ):
+        raise ValueError('input-too-long')
+
+
+def infer(request, manifest):
+    weights = verify_artifacts(manifest)
+    import torch
+    from cua_s1.model import ChoiceExample, load_checkpoint
+
+    torch.set_num_threads(1)
+    torch.use_deterministic_algorithms(True)
+    model, collator, config = load_checkpoint(weights, 'cpu')
+    check_limits(request, config)
+    example = ChoiceExample(context=request['context'], options=tuple(o['text'] for o in request['options']), label=0)
+    with torch.inference_mode():
+        probabilities = model(collator([example])).softmax(-1)[0].tolist()
+    return {
+        'schemaVersion': 1, 'requestHash': request['requestHash'], 'catalogHash': request['catalogHash'],
+        'model': {'id': manifest['modelId'], 'revision': manifest['revision'], 'domain': manifest['domain']},
+        'scores': [{'id': option['id'], 'probability': probability} for option, probability in zip(request['options'], probabilities, strict=True)],
+    }
+
+
+def source_revision(direct_url_text):
+    """Return the PEP 610 VCS commit recorded at install time, or None when unprovable."""
+    try:
+        commit = json.loads(direct_url_text)['vcs_info']['commit_id']
+    except Exception:
+        return None
+    return commit if isinstance(commit, str) and re.fullmatch(r'[0-9a-f]{40}', commit) else None
+
+
+def provenance():
+    """Installed runtime/source facts only; never imports torch or loads a checkpoint."""
+    import importlib.metadata as metadata
+    import platform
+
+    cua = {'distribution': None, 'version': None, 'sourceRevision': None}
+    names = sorted(set(metadata.packages_distributions().get('cua_s1', [])))
+    if len(names) == 1:
+        dist = metadata.distribution(names[0])
+        cua = {'distribution': names[0], 'version': dist.version, 'sourceRevision': source_revision(dist.read_text('direct_url.json'))}
+    try:
+        torch = metadata.version('torch')
+    except metadata.PackageNotFoundError:
+        torch = None
+    return {'schemaVersion': 1, 'python': {'implementation': platform.python_implementation(), 'version': platform.python_version()},
+            'cuaS1': cua, 'torch': torch}
+
+
+def main():
+    try:
+        if sys.argv[1:] == ['--provenance']:
+            print(json.dumps(provenance(), allow_nan=False))
+            return 0
+        manifest = json.loads(sys.argv[1])
+        raw = sys.stdin.buffer.read(2 * 1024 * 1024 + 1)
+        if len(raw) > 2 * 1024 * 1024:
+            raise ValueError('request-limit')
+        request = json.loads(raw)
+        result = infer(request, manifest)
+        print(json.dumps(result, allow_nan=False))
+    except Exception:
+        # No raw prompt, selected values, artifact paths, or traceback on the wire.
+        print('cua-provider-unavailable', file=sys.stderr)
+        return 1
+    return 0
+
+
+if __name__ == '__main__':
+    sys.exit(main())

package/harness-everything/scripts/system-one/evaluate.js

@@ -0,0 +1,98 @@
+'use strict';
+const { validateRequest } = require('./contract');
+const { TIER_OPTIONS } = require('./router');
+const exact = (obj, keys) => obj && typeof obj === 'object' && !Array.isArray(obj) && Object.keys(obj).length === keys.length && keys.every(k => Object.hasOwn(obj, k));
+const nonempty = v => typeof v === 'string' && v.trim().length > 0;
+const labels = ['tier1', 'tier2', 'tier3', null];
+const finite = v => typeof v === 'number' && Number.isFinite(v);
+const fail = () => { throw new TypeError('invalid-evaluation-input'); };
+function validateCorpus(corpus) {
+  if (!exact(corpus, ['schemaVersion', 'cases']) || corpus.schemaVersion !== 1 || !Array.isArray(corpus.cases) || !corpus.cases.length) fail();
+  const ids = new Set(); const hashes = new Set(); const families = new Map();
+  for (const c of corpus.cases) {
+    if (!exact(c, ['id', 'family', 'split', 'language', 'source', 'reviewed', 'request', 'gold'])
+      || ![c.id, c.family, c.source].every(nonempty) || !['train', 'validation', 'holdout'].includes(c.split)
+      || !['en', 'zh-TW'].includes(c.language) || typeof c.reviewed !== 'boolean' || !labels.includes(c.gold)) fail();
+    validateRequest(c.request);
+    if (c.request.task !== 'tier' || JSON.stringify(c.request.options) !== JSON.stringify(TIER_OPTIONS)
+      || ids.has(c.id) || hashes.has(c.request.requestHash) || (families.has(c.family) && families.get(c.family) !== c.split)) fail();
+    ids.add(c.id); hashes.add(c.request.requestHash); families.set(c.family, c.split);
+  }
+  return true;
+}
+function prediction(decision) {
+  if (exact(decision, ['status', 'reason']) && decision.status === 'unavailable' && nonempty(decision.reason)) return null;
+  if (!exact(decision, ['status', 'reason', 'selectedId', 'confidence', 'margin', 'model'])
+    || !['accepted', 'abstain', 'invalid-output'].includes(decision.status) || !nonempty(decision.reason)
+    || !labels.includes(decision.selectedId) || (decision.status === 'accepted') !== (decision.selectedId !== null)) fail();
+  for (const key of ['confidence', 'margin']) if (decision[key] !== null && (!finite(decision[key]) || decision[key] < 0 || decision[key] > 1)) fail();
+  if (decision.model !== null && (!exact(decision.model, ['id', 'revision', 'domain']) || !Object.values(decision.model).every(nonempty))) fail();
+  if (decision.status === 'accepted' && (decision.confidence === null || decision.margin === null || decision.model?.domain !== 'harness-routing-v1')) fail();
+  return decision.selectedId;
+}
+// Scored decisions carry the full catalog-order probability vector; unavailable/invalid outputs carry null.
+function checkScores(decision, scores) {
+  const scored = ['accepted', 'abstain'].includes(decision.status);
+  if (!scored) { if (scores !== null) fail(); return; }
+  if (!Array.isArray(scores) || scores.length !== TIER_OPTIONS.length
+    || scores.some(p => !finite(p) || p < 0 || p > 1) || Math.abs(scores.reduce((s, p) => s + p, 0) - 1) > 1e-6) fail();
+  const ranked = [...scores].sort((a, b) => b - a);
+  if (decision.confidence !== ranked[0] || decision.margin !== ranked[0] - ranked[1]) fail();
+}
+function metrics(gold, predicted) {
+  const count = gold.length;
+  const accepted = predicted.filter(x => x !== null).length;
+  const correct = predicted.filter((x, i) => x === gold[i]).length;
+  const acceptedCorrect = predicted.filter((x, i) => x !== null && x === gold[i]).length;
+  const macroF1 = labels.reduce((sum, label) => {
+    const tp = gold.filter((g, i) => g === label && predicted[i] === label).length;
+    const fp = gold.filter((g, i) => g !== label && predicted[i] === label).length;
+    const fn = gold.filter((g, i) => g === label && predicted[i] !== label).length;
+    return sum + (2 * tp + fp + fn ? 2 * tp / (2 * tp + fp + fn) : 0);
+  }, 0) / labels.length;
+  return { accuracy: correct / count, macroF1, coverage: accepted / count, abstentionRate: 1 - accepted / count,
+    acceptedPrecision: accepted ? acceptedCorrect / accepted : null,
+    acceptedErrorRate: accepted ? (accepted - acceptedCorrect) / accepted : null };
+}
+function p95(values) { return values.length ? [...values].sort((a, b) => a - b)[Math.ceil(values.length * 0.95) - 1] : null; }
+// sourceEvidence is the provider provenance result; only the pinned cua_s1 revision satisfies the gate.
+function evaluate(corpus, records, sourceEvidence = null) {
+  validateCorpus(corpus);
+  const cases = corpus.cases.filter(c => c.split === 'holdout');
+  if (!cases.length || !Array.isArray(records) || records.length !== cases.length) fail();
+  const byId = new Map(); const models = new Set(); const cold = []; const warm = [];
+  for (const record of records) {
+    if (!exact(record, ['id', 'baseline', 'runs']) || byId.has(record.id) || !cases.some(c => c.id === record.id)
+      || !labels.includes(record.baseline) || !Array.isArray(record.runs) || record.runs.length !== 2) fail();
+    for (const run of record.runs) {
+      if (!exact(run, ['decision', 'scores', 'latencyMs', 'coldStart']) || !finite(run.latencyMs) || run.latencyMs < 0 || typeof run.coldStart !== 'boolean') fail();
+      prediction(run.decision);
+      checkScores(run.decision, run.scores);
+      if (run.decision.model) models.add(JSON.stringify([run.decision.model.id, run.decision.model.revision, run.decision.model.domain]));
+      (run.coldStart ? cold : warm).push(run.latencyMs);
+    }
+    byId.set(record.id, record);
+  }
+  if (models.size > 1) fail();
+  const gold = cases.map(c => c.gold);
+  const baseline = metrics(gold, cases.map(c => byId.get(c.id).baseline));
+  const model = metrics(gold, cases.map(c => prediction(byId.get(c.id).runs[0].decision)));
+  const canonicalRun = ({ decision: d, scores }) => JSON.stringify([d.status, d.reason, d.selectedId ?? null, d.confidence ?? null, d.margin ?? null,
+    d.model ? [d.model.id, d.model.revision, d.model.domain] : null, scores]);
+  const agreement = records.filter(r => canonicalRun(r.runs[0]) === canonicalRun(r.runs[1])).length / cases.length;
+  const latency = { coldP95Ms: p95(cold), warmP95Ms: p95(warm), coldSamples: cold.length, warmSamples: warm.length };
+  const gates = {
+    reviewedHoldout: cases.length >= 200 && cases.every(c => c.reviewed)
+      && ['en', 'zh-TW'].every(lang => cases.filter(c => c.language === lang).length >= 50),
+    acceptedPrecision: model.acceptedPrecision !== null && model.acceptedPrecision >= 0.98,
+    coverage: model.coverage >= 0.8,
+    macroF1: model.macroF1 >= baseline.macroF1,
+    repeatability: agreement === 1,
+    warmLatency: warm.length >= cases.length && latency.warmP95Ms <= 100,
+    sourceProvenance: sourceEvidence?.status === 'recorded' && sourceEvidence.provenance?.cuaS1?.sourceRevisionStatus === 'pinned',
+    policyEvidence: false,
+    liveHostEvidence: false,
+  };
+  return { schemaVersion: 1, cases: cases.length, baseline, model, agreement, latency, gates, rolloutReady: Object.values(gates).every(Boolean) };
+}
+module.exports = { validateCorpus, evaluate };

package/harness-everything/scripts/system-one/install.js

@@ -0,0 +1,152 @@
+#!/usr/bin/env node
+'use strict';
+// Opt-in System One installer: a dedicated venv, pinned cua_s1 source, and a pinned CPU checkpoint.
+// It is never run from hooks, and it never changes the router mode; it prints the environment to set.
+const fs = require('node:fs');
+const os = require('node:os');
+const path = require('node:path');
+const { createHash } = require('node:crypto');
+const { spawnSync } = require('node:child_process');
+const { performance } = require('node:perf_hooks');
+const { validateManifest, provenance, score, PINNED_CUA_S1_REVISION } = require('./provider');
+const { createRequest, decide } = require('./contract');
+const { TIER_OPTIONS } = require('./router');
+
+const PINS = Object.freeze({
+  cuaS1Revision: PINNED_CUA_S1_REVISION,
+  cuaS1Spec: `cua-s1 @ git+https://github.com/trycua/cua@${PINNED_CUA_S1_REVISION}#subdirectory=libs/cua-s1/python`,
+  torchSpec: 'torch>=2.2,<3',
+  torchIndex: 'https://download.pytorch.org/whl/cpu',
+  modelId: 'cua-ai/cua-s1-forms',
+  modelRevision: 'f54adbf447f4ca6ec259f529ee3f2e3e09f8cc71',
+  domain: 'forms-v1',
+  // sha256/size from the Hugging Face tree API at modelRevision (LFS oid for weights; the JSON
+  // sidecar's git blob id d8f8426a... was re-hashed to confirm these exact CRLF bytes).
+  files: Object.freeze({
+    weights: Object.freeze({ name: 'cua-s1-forms.safetensors', size: 2828784, sha256: '05954c1caf51c2fb6c13ea4acbfc88a2e7653dea192252bb51dc89e76a356ddc' }),
+    config: Object.freeze({ name: 'cua-s1-forms.json', size: 1053, sha256: '62d31e2f9a001a8e9b6f8534c5194d07ebdd3f9d62ef1ac281906622992650ca' }),
+  }),
+});
+const USAGE = 'Usage: install.js [--dir <absolute dir>] [--python <3.11-3.13 executable>] [--dry-run]';
+
+const defaultDir = () => path.join(os.homedir(), '.agents', 'harness-everything', 'system-one');
+const fileUrl = name => `https://huggingface.co/${PINS.modelId}/resolve/${PINS.modelRevision}/${name}`;
+const checkpointDir = dir => path.join(dir, 'checkpoints', PINS.modelRevision);
+const venvPython = (dir, platform = process.platform) => (platform === 'win32'
+  ? path.join(dir, 'venv', 'Scripts', 'python.exe') : path.join(dir, 'venv', 'bin', 'python'));
+function supportedPython(version) {
+  const match = typeof version === 'string' && /^3\.(\d+)\.\d+$/.exec(version);
+  return Boolean(match) && Number(match[1]) >= 11 && Number(match[1]) <= 13;
+}
+function pythonCandidates(platform = process.platform) {
+  return platform === 'win32'
+    ? [['py', '-3.13'], ['py', '-3.12'], ['py', '-3.11'], ['python'], ['python3']]
+    : [['python3.13'], ['python3.12'], ['python3.11'], ['python3'], ['python']];
+}
+function parseArgs(argv) {
+  const opts = { dir: defaultDir(), python: null, dryRun: false };
+  const seen = new Set();
+  for (let i = 0; i < argv.length; i++) {
+    const flag = argv[i];
+    if (seen.has(flag)) throw new Error('usage');
+    seen.add(flag);
+    if (flag === '--dry-run') opts.dryRun = true;
+    else if (flag === '--dir' || flag === '--python') {
+      const value = argv[++i];
+      if (!value || value.startsWith('--')) throw new Error('usage');
+      if (flag === '--dir' && !path.isAbsolute(value)) throw new Error('usage');
+      opts[flag.slice(2)] = flag === '--dir' ? path.resolve(value) : value;
+    } else throw new Error('usage');
+  }
+  return opts;
+}
+function buildManifest(dir, python) {
+  const manifest = { schemaVersion: 1, python, checkpoint: path.join(checkpointDir(dir), PINS.files.weights.name),
+    weightsSha256: PINS.files.weights.sha256, configSha256: PINS.files.config.sha256,
+    modelId: PINS.modelId, revision: PINS.modelRevision, domain: PINS.domain,
+    // Each score starts a fresh Python process that imports torch; cold start dominates latency.
+    timeoutMs: 10000 };
+  validateManifest(manifest);
+  return manifest;
+}
+const matches = (buf, spec) => buf.length === spec.size && createHash('sha256').update(buf).digest('hex') === spec.sha256;
+async function httpFetch(url) {
+  const res = await fetch(url, { redirect: 'follow' });
+  if (!res.ok) throw new Error(`download-failed:${res.status}`);
+  return Buffer.from(await res.arrayBuffer());
+}
+async function ensureFile(url, dest, spec, fetcher = httpFetch) {
+  if (fs.existsSync(dest) && matches(fs.readFileSync(dest), spec)) return 'verified-existing';
+  const bytes = await fetcher(url);
+  if (!Buffer.isBuffer(bytes) || !matches(bytes, spec)) throw new Error(`download-verification:${spec.name}`);
+  const tmp = `${dest}.${process.pid}.tmp`;
+  fs.writeFileSync(tmp, bytes);
+  fs.renameSync(tmp, dest);
+  return 'downloaded';
+}
+function exec(cmd, args, capture = false) {
+  const result = spawnSync(cmd, args, { shell: false, windowsHide: true, encoding: 'utf8', stdio: capture ? 'pipe' : 'inherit' });
+  if (result.error || result.status !== 0) throw new Error(`command-failed: ${[cmd, ...args].join(' ')}`);
+  return capture ? result.stdout.trim() : '';
+}
+const VERSION_PROBE = ['-c', 'import sys; print("%d.%d.%d" % sys.version_info[:3])'];
+function findPython(explicit) {
+  for (const [cmd, ...pre] of explicit ? [[explicit]] : pythonCandidates()) {
+    const result = spawnSync(cmd, [...pre, ...VERSION_PROBE], { shell: false, windowsHide: true, encoding: 'utf8' });
+    const version = result.status === 0 ? result.stdout.trim() : null;
+    if (supportedPython(version)) return { cmd, pre, version };
+  }
+  throw new Error(explicit ? 'python-unsupported: need Python 3.11-3.13' : 'python-not-found: install Python 3.11-3.13 or pass --python');
+}
+function plan(opts) {
+  const venvPy = venvPython(opts.dir);
+  const manifestPath = path.join(opts.dir, 'manifest.json');
+  return { dir: opts.dir, manifest: manifestPath,
+    steps: [
+      `create venv ${path.join(opts.dir, 'venv')} with ${opts.python || 'first Python 3.11-3.13 found'}`,
+      `${venvPy} -m pip install --index-url ${PINS.torchIndex} "${PINS.torchSpec}"`,
+      `${venvPy} -m pip install "${PINS.cuaS1Spec}"`,
+      ...Object.values(PINS.files).map(f => `download ${fileUrl(f.name)} (sha256 ${f.sha256})`),
+      `write ${manifestPath}`,
+      'verify provenance (pinned revision) and one real CPU inference',
+    ],
+    env: { HARNESS_SYSTEM_ONE_MODE: 'shadow', HARNESS_SYSTEM_ONE_CONFIG: manifestPath } };
+}
+function verify(manifestPath) {
+  const source = provenance(manifestPath);
+  const request = createRequest('tier', 'fix a typo in README', TIER_OPTIONS);
+  const start = performance.now();
+  const result = score(request, manifestPath);
+  const latencyMs = Math.round(performance.now() - start);
+  const decision = result.status === 'scored' ? decide(request, result.response) : result;
+  const scores = result.status === 'scored' && decision.status !== 'invalid-output' ? result.response.scores : null;
+  const ok = source.status === 'recorded' && source.provenance.cuaS1.sourceRevisionStatus === 'pinned' && scores !== null;
+  return { ok, source, decision, scores, latencyMs };
+}
+async function main(argv) {
+  const opts = parseArgs(argv);
+  const p = plan(opts);
+  if (opts.dryRun) { process.stdout.write(`${JSON.stringify({ dryRun: true, ...p }, null, 2)}\n`); return 0; }
+  const base = findPython(opts.python);
+  const venvPy = venvPython(opts.dir);
+  fs.mkdirSync(checkpointDir(opts.dir), { recursive: true });
+  if (!fs.existsSync(venvPy)) exec(base.cmd, [...base.pre, '-m', 'venv', path.join(opts.dir, 'venv')]);
+  const venvVersion = exec(venvPy, VERSION_PROBE, true);
+  if (!supportedPython(venvVersion)) throw new Error(`python-unsupported: existing venv uses ${venvVersion}`);
+  exec(venvPy, ['-m', 'pip', 'install', '--disable-pip-version-check', '--index-url', PINS.torchIndex, PINS.torchSpec]);
+  exec(venvPy, ['-m', 'pip', 'install', '--disable-pip-version-check', PINS.cuaS1Spec]);
+  for (const spec of Object.values(PINS.files)) {
+    console.error(`${spec.name}: ${await ensureFile(fileUrl(spec.name), path.join(checkpointDir(opts.dir), spec.name), spec)}`);
+  }
+  fs.writeFileSync(p.manifest, `${JSON.stringify(buildManifest(opts.dir, venvPy), null, 2)}\n`);
+  const check = verify(p.manifest);
+  process.stdout.write(`${JSON.stringify({ dryRun: false, ...p, verification: check }, null, 2)}\n`);
+  return check.ok ? 0 : 1;
+}
+if (require.main === module) {
+  main(process.argv.slice(2)).then(code => { process.exitCode = code; }, err => {
+    if (err.message === 'usage') { console.error(USAGE); process.exitCode = 2; return; }
+    console.error(`System One install failed: ${err.message}`); process.exitCode = 1;
+  });
+}
+module.exports = { PINS, fileUrl, venvPython, supportedPython, pythonCandidates, parseArgs, buildManifest, ensureFile, plan, verify };