harness-everything 0.21.0 → 0.22.1

package/benchmarks/results/system-one/pre-merge-review-2026-09-23/README.md

@@ -0,0 +1,92 @@
+# System One: pre-merge review of the resident provider (2026-09-23)
+
+This is the retained evidence behind PR [#240](https://github.com/dyphn1/Harness-everything/pull/240) for issue [#233](https://github.com/dyphn1/Harness-everything/issues/233).
+
+- **Evidence layers:** independent lane reports, local real-checkpoint runs, and one live-host observation.
+- **Not claimed:** any Harness routing quality; the only available checkpoint is `cua-ai/cua-s1-forms` (domain `forms-v1`).
+- **Redaction:** lane reports were copied verbatim except for secrets, user names and local paths. Tokens are `<redacted>`; paths are `~`, `<repo>` or `<lane-worktree>`.
+
+## Files
+
+| File | What it is |
+| --- | --- |
+| [haiku-spec-conformance.md](haiku-spec-conformance.md) | Claude Haiku lane: clause-by-clause spec check against the real CPU checkpoint |
+| [luna-adversarial.md](luna-adversarial.md) | Codex `gpt-5.6-luna` lane: adversarial security/robustness findings F-01–F-05 |
+| [skill-top1-probe.js](skill-top1-probe.js) | Script for the historical-keyword → skill top-1 probe |
+| [skill-top1-probe.json](skill-top1-probe.json) | Unmodified probe output: option catalog, per-input rows, summary |
+
+The lanes ran in separate git worktrees at `42655fa`. Each used its own manifest copy (real venv and checkpoint), so each got its own resident server. Neither lane was allowed to edit product code.
+
+## What was re-verified by the reviewer (lane claims are not accepted alone)
+
+- **Haiku, 24/24 PASS:** I re-ran its idle-exit script. With `idleTimeoutMs: 60000`, the state file was present at 55 s and gone at 60 s.
+  - Discrepancy: the report names `.lane/idle/manifest.json`, but the script used `.lane/idle-manifest.json`. The result is unaffected.
+- **Luna, F-01–F-05:** each finding got a RED-first regression test in PR #240 (commits `ee5ab0f`, `e59af73`). I then re-ran the lane's own probe scripts against the fixed code with the real model:
+
+| # | Before | After |
+| --- | --- | --- |
+| F-01 | state file (token) inherited `Users:(RX)` / `Authenticated Users:(M)` on D:\\ | protected DACL with one full-control ACE for the current user SID |
+| F-02 | a silent connection made a legitimate call time out (1252 ms) | legitimate call scored in 305 ms |
+| F-03 | corrupted state left the old server alive and serving | old server exits (`oldPidAlive: false`) |
+| F-04 | un-spawnable executable left a plain 60 s lock | `spawn-failed` → `provider-unavailable` |
+| F-05 | 257 options accepted | `invalid-request` |
+
+- **CI caught two follow-ups on the first push:**
+  - The GitHub Windows runner creates files with *explicit* SYSTEM, Administrators and OWNER RIGHTS entries that `icacls /grant:r` keeps. The whole DACL is now replaced through `SetNamedSecurityInfoW`.
+  - On POSIX, an exited child stays a zombie until the blocked test process reaps it, so S1-RS09 now asserts "stopped serving".
+- **F-03 in real use:** before the fix, this machine had an orphan server for the real manifest that its state file no longer named. It was most likely produced when a burst overflowed the old backlog of 16 and the client treated the refused connection as a dead server. I stopped it manually.
+
+## Install forms
+
+The hook entry `kernel-router.js` was run with a host-shaped stdin payload (`{"prompt": ...}`) in shadow mode against the real resident server. Every form below scored (`abstain/domain-mismatch`) with no token in the output:
+
+- repo checkout
+- staged Claude plugin (tracked files only)
+- Codex plugin copy
+- skills install `--claude --codex --copy` into a temp project
+- after release: the installed Claude Code 0.22.0 and Codex 0.22.0 plugin caches
+
+The public OpenAI Skills-only bundle ships the scripts but no hooks.
+
+- **Packaging defect found here:** `__pycache__/*.pyc` from Python test runs was being shipped by plugin sync, the skills installer and the public submission bundle. Fixed in PR #240 (`58135ae`).
+- **Cold start through the hook entry:** returned `provider-starting` in 473 ms. The detached server survived the hook process exiting, and the next call was scored.
+
+## Live-host observation (Windows 11, Claude Code, plugin 0.22.0)
+
+Setup:
+- `~/.claude/settings.json` `env` set to `HARNESS_SYSTEM_ONE_MODE=shadow` plus the manifest path.
+- The resident server stopped first.
+
+Then one headless `claude -p "Reply with the single word OK." --max-turns 1` session ran:
+1. The session exited after 8 s with output `OK`.
+2. Twelve seconds later a resident server was running, with a start time inside that session's window.
+
+Conclusion: the real `UserPromptSubmit` hook started the server, and the server outlived the host's hook runner.
+
+- **Scope:** one observation. macOS/Linux hosts and a Codex host were not live-verified.
+- **Host environment variables:** settings `env` values are hot-reloaded into the session and are inherited by every command the agent runs, not only by hooks. A Codex started from a normal terminal sees them only if they are set at user/OS level.
+
+## Historical keywords → does top-1 point to the right skill?
+
+Method:
+- Candidates are the 26 canonical skills. Each option is `"<skill>: <SKILL.md description>"`, cut at a word boundary to ≤ 96 bytes (the checkpoint's option limit).
+- Set A: each `routing-keywords.json` guide-group keyword. Gold = the skills its group routes to.
+- Set B: each positive `evals/*/tasks/*.yaml` prompt. Gold = that eval's skill.
+- The raw argmax is reported. `decide()` rejected every result as `domain-mismatch`.
+
+| Set | n | Model top-1 | Random top-1 | Model top-3 | Random top-3 |
+| --- | --- | --- | --- | --- | --- |
+| A: keywords | 305 | 8.2% | 11.6% | 25.9% | 31.0% |
+| B: eval prompts | 34 | 5.9% | 3.8% | 8.8% | 11.5% |
+
+- **Collapse:** top-1 piles onto a few options; in set A, `find-skills` takes 90 of 305.
+- **High-confidence errors:** 16 in set A and 1 in set B have p ≥ 0.9, for example `refactor` → `skill-creator` at 0.95.
+- **Lexical baseline:** the lexical router recommends the gold skill for 29 of 34 set-B prompts.
+- **Conclusion:** the forms checkpoint cannot route skills. Keep System One `off` until a Harness-trained checkpoint passes the Phase 4 gates.
+
+Re-run (needs the installed manifest):
+
+```bash
+node benchmarks/results/system-one/pre-merge-review-2026-09-23/skill-top1-probe.js \
+  "$(pwd)" ~/.agents/harness-everything/system-one/manifest.json /tmp/skill-top1-probe.json
+```

package/benchmarks/results/system-one/pre-merge-review-2026-09-23/haiku-spec-conformance.md

@@ -0,0 +1,241 @@
+# PR #240 Spec Conformance Test Report
+## Resident System One Provider (Phase 4 prerequisite)
+
+**Test Date:** 2026-09-23  
+**Worktree:** <lane-worktree>  
+**Node.js:** v24.18.0  
+**Real Model:** CUA-S1-FORMS via resident Python server  
+**Manifest:** .lane/manifest.json (transport: "resident", idleTimeoutMs: 1800000)
+
+---
+
+## Test Results Summary
+
+| # | Spec Clause (Quote) | Command | Expected | Observed | Status |
+|---|---|---|---|---|---|
+| 1.1 | **Cold start - immediate return** "If no live server exists, the Node provider creates a start lock with exclusive-create, spawns `cua_adapter.py --serve` detached... and returns `unavailable/provider-starting`" | `provider.score(createRequest('tier', 'fix a typo in README', TIER_OPTIONS), MANIFEST_PATH)` with no server running | Returns {status:'unavailable', reason:'provider-starting'} in <1s | Returned {status:'unavailable', reason:'provider-starting'} in 38ms | **PASS** |
+| 1.2 | **Cold start - lock file** "creates a start lock with exclusive-create" | Check .starting lock file exists after score() call | Lock file appears at `system-one-resident-<digest>.starting` | Lock file created at expected path | **PASS** |
+| 1.3 | **Cold start - state file** "The server verifies artifact hashes before `load_checkpoint`, writes its state file atomically" | Monitor .lane/ for state file appearance | State file `system-one-resident-<digest>.json` appears within 60s | State file appeared at ~4100ms | **PASS** |
+| 1.4 | **Cold start - lock removal** "writes its state file atomically, then removes the lock" | Check .starting file removed | Lock file disappears once state file is written | Lock removed after state file created | **PASS** |
+| 1.5 | **Cold start - second call scored** "a later call returns status 'scored'" | Call provider.score() after server ready (~6s later) | Returns {status:'scored', response:{...}} with valid Phase 0 response | Returned scored response with full probability vector | **PASS** |
+| 1.6 | **Cold start - domain-mismatch** "whose decision via decide() is 'abstain'/'domain-mismatch'" | Call decide() on scored response | Decision has reason='domain-mismatch' (forms-v1 domain mismatch in harness-routing-v1 context) | reason: "domain-mismatch", confidence: 0.995, margin: 0.991 | **PASS** |
+| 2.1 | **State file format - fields** "The state file has exactly `schemaVersion: 1`, `pid`, `port`, `token` (64 hex), and `startedAt`" | Read .lane/system-one-resident-*.json | JSON with exactly those 5 keys, schemaVersion=1, token matches /^[0-9a-f]{64}$/ | {schemaVersion:1, pid:7804, port:59489, token:"90f1..." (64 hex), startedAt:1790136963.824} | **PASS** |
+| 2.2 | **State file - key count** "has exactly" (5 keys) | Object.keys(state).length | Must equal 5 | Count: 5 keys (pid, port, schemaVersion, startedAt, token) | **PASS** |
+| 2.3 | **State file - token format** "token (64 hex)" | Validate token: /^[0-9a-f]{64}$/ | Matches 64 lowercase hex digits | "<redacted-token>" ✓ | **PASS** |
+| 3.1 | **Shadow mode - diagnostic** "When enabled, a single `SYSTEM ONE` JSON diagnostic reports mode, status, reason, selected candidate, model identity, confidence and margin" | Set HARNESS_SYSTEM_ONE_MODE=shadow, run kernel-router-core.js | Output contains line: `=> SYSTEM ONE: {...}` with JSON structure | Found: `=> SYSTEM ONE: {"mode":"shadow","status":"abstain","reason":"domain-mismatch",...}` | **PASS** |
+| 3.2 | **Shadow mode - no token leak** "it omits prompt text, score vectors and artifact paths" | Check SYSTEM ONE JSON line for token | Token must not appear in diagnostic JSON | No 64-hex token in SYSTEM ONE line; only in state file | **PASS** |
+| 3.3 | **Shadow mode - no prompt leak** | Check SYSTEM ONE JSON line for prompt text | Prompt must not appear in diagnostic JSON | Prompt "fix a typo in README" not in SYSTEM ONE line | **PASS** |
+| 3.4 | **Shadow mode - identical plan** "the \"ROUTER WORKFLOW PLAN (JSON)\" line is identical to mode off" | Compare router output with mode=off vs mode=shadow | Same ROUTER WORKFLOW PLAN JSON block | Plan identical in both modes, tier1, direct-single strategy | **PASS** |
+| 4.1 | **Idle exit - timeout** "The server exits after `idleTimeoutMs` without requests" | Create manifest with idleTimeoutMs:60000, start, wait 75s without calls | State file removed at ~60s mark | State file existed through 55s, gone by 60s | **PASS** |
+| 4.2 | **Idle exit - state removal** "exits... deleting only its own state file" | Monitor .lane/system-one-resident-*.json during idle timeout | State file deleted, lock file (if any) also gone | State file removed after idle timeout | **PASS** |
+| 5.1 | **Hard-kill recovery - detection** "refused connection or dead PID → stale state removed and a new start (`provider-starting`)" | Kill server PID, call provider.score() | Returns {status:'unavailable', reason:'provider-starting'} | Called after hard kill returned provider-starting | **PASS** |
+| 5.2 | **Hard-kill recovery - state cleanup** | After kill, check state file removed | State file should be gone before next call returns | State file removed after detecting dead process | **PASS** |
+| 5.3 | **Hard-kill recovery - respawn** "a new start" | Call provider.score() again after recovery wait | Returns {status:'scored', response:{...}} with valid response | After 10s wait, score() returned scored response | **PASS** |
+| 6.1 | **Stop CLI - removes state** "node harness-everything/scripts/system-one/resident.js stop" | Run: resident.js stop /absolute/manifest.json | State file removed, command returns true | State file deleted, command output: {running:false} | **PASS** |
+| 6.2 | **Stop CLI - status false** "status reports running:false" | Call resident.status() after stop | Returns {running:false} | {running:false} confirmed | **PASS** |
+| 6.3 | **Stop CLI - no token in output** "CLI output never contains the token" | Capture stdout/stderr of stop command | No 64-hex token in output | Output: {"running":false}, no token visible | **PASS** |
+| 7.1 | **Installer dry-run - valid JSON** "`--dry-run` prints the plan without changes; stdout is valid JSON" | Run: install.js --dir .lane/inst --dry-run | Valid JSON output, contains 'dryRun':true | Valid JSON: {dryRun:true, dir:"...", steps:[...], env:{...}} | **PASS** |
+| 7.2 | **Installer dry-run - transport mention** "mentions transport-resident verification" | Check output for transport/resident references | Plan steps mention "verify provenance" and resident server | Step: "verify provenance (pinned revision), start the resident server, one warm CPU inference, stop it" | **PASS** |
+| 7.3 | **Installer dry-run - creates nothing** "creates nothing" | Check file system after --dry-run | No files/directories created in .lane/inst | Directory .lane/inst not created (confirmed: false) | **PASS** |
+| 8.1 | **Warm latency - p50** "time 30 sequential provider.score calls; report p50/p95" | Run 30 sequential score calls on ready server | p50 latency (milliseconds) | p50: 49ms (min:36ms, max:98ms, avg:59.43ms) | **PASS** |
+| 8.2 | **Warm latency - p95** | Same as 8.1 | p95 latency (milliseconds) | p95: 93ms | **PASS** |
+
+---
+
+## Detailed Test Evidence
+
+### Test 1: Cold Start Behavior
+**Spec requirement:** "A resident-mode score never blocks on model loading. If no live server exists, the Node provider creates a start lock with exclusive-create, spawns `cua_adapter.py --serve` detached with the configured Python (no shell), and returns `unavailable/provider-starting`"
+
+**Execution:**
+```javascript
+const provider = require('./provider');
+const result = provider.score(createRequest('tier', 'fix a typo in README', TIER_OPTIONS), MANIFEST_PATH);
+// With no server running
+```
+
+**Observations:**
+- First call returned `{status:'unavailable', reason:'provider-starting'}` in 38ms ✓
+- Lock file appeared: `system-one-resident-2764d6fab6feb5d6.starting` ✓
+- State file appeared: `system-one-resident-2764d6fab6feb5d6.json` at ~4100ms ✓
+- Lock file removed after state written ✓
+- Second call (after 6s wait for server startup) returned `{status:'scored', response:{...}}` ✓
+- Scores summed to 1.0, probabilities in [0,1] ✓
+- Model domain: "forms-v1" (expected for this checkpoint) ✓
+- Decision via decide(): reason='domain-mismatch' (correct: forms-v1 vs harness-routing-v1 policy domain) ✓
+
+### Test 2: State File Format
+**Spec requirement:** "The state file has exactly `schemaVersion: 1`, `pid`, `port`, `token` (64 hex), and `startedAt`, and is created with mode 0600"
+
+**Observed state file content:**
+```json
+{
+  "schemaVersion": 1,
+  "pid": 7804,
+  "port": 59489,
+  "token": "<redacted>",
+  "startedAt": 1790136963.824462
+}
+```
+
+**Validation:**
+- Exactly 5 fields ✓
+- schemaVersion: 1 ✓
+- pid: positive integer ✓
+- port: valid range 1-65535 ✓
+- token: 64 lowercase hex characters ✓
+- startedAt: finite number (Unix timestamp) ✓
+
+### Test 3: Shadow Mode Diagnostics
+**Spec requirement:** "When enabled, a single `SYSTEM ONE` JSON diagnostic reports mode, status, reason, selected candidate, model identity, confidence and margin; it omits prompt text, score vectors and artifact paths"
+
+**Observed diagnostic:**
+```
+=> SYSTEM ONE: {"mode":"shadow","status":"abstain","reason":"domain-mismatch","selectedId":null,"confidence":0.995292067527771,"margin":0.9907588204368949,"model":{"id":"cua-ai/cua-s1-forms","revision":"f54adbf447f4ca6ec259f529ee3f2e3e09f8cc71","domain":"forms-v1"},"applied":false}
+```
+
+**Security validation:**
+- Prompt "fix a typo in README" NOT in diagnostic ✓
+- Token NOT in diagnostic ✓
+- Score vectors NOT in diagnostic ✓
+- Artifact paths NOT in diagnostic ✓
+- Mode: "shadow" reported correctly ✓
+- Model identity included ✓
+- Confidence and margin included ✓
+
+**Router plan comparison:**
+- ROUTER WORKFLOW PLAN (JSON) identical whether mode=off or mode=shadow ✓
+
+### Test 4: Idle Exit Timeout
+**Spec requirement:** "The server exits after `idleTimeoutMs` without requests, on `shutdown`, or when either checkpoint file's size or modification time changes (`artifact-changed`), deleting only its own state file"
+
+**Test configuration:**
+- Created second manifest with `idleTimeoutMs: 60000`
+- Started server, confirmed state file present
+- Waited 75 seconds without making any score calls
+- Monitored state file existence
+
+**Observations:**
+- Server held state file through 55s: ✓
+- State file disappeared between 55s and 60s (exact timeout window) ✓
+- By 60s mark: state file gone, not recreated ✓
+- No lock file left behind ✓
+
+### Test 5: Server Hard-Kill Recovery
+**Spec requirement:** "refused connection or dead PID → stale state removed and a new start (`provider-starting`)"
+
+**Test sequence:**
+1. Start server normally
+2. Extract PID from state file
+3. Force kill process: `process.kill(pid, 'SIGKILL')`
+4. Call provider.score() immediately
+5. Wait 10s for recovery
+6. Call provider.score() again
+
+**Observations:**
+- After kill, score() called returned: `{status:'unavailable', reason:'provider-starting'}` ✓
+- State file removed before restart (pidAlive check detected dead process) ✓
+- New lock file created ✓
+- After 10s, server recovered and returned: `{status:'scored', response:{...}}` ✓
+
+### Test 6: Stop via CLI
+**Spec requirement:** "node harness-everything/scripts/system-one/resident.js stop <absolute manifest> removes the state file; status reports running:false; CLI output never contains the token"
+
+**Commands executed:**
+```bash
+node resident.js start /absolute/manifest.json
+# ... (server running) ...
+node resident.js stop /absolute/manifest.json
+node resident.js status /absolute/manifest.json
+```
+
+**Observations:**
+- Start command: output showed {running:true, pid:..., port:..., model:...} ✓
+- Stop command: output showed {running:false} ✓
+- State file removed after stop ✓
+- No 64-hex token in any command output ✓
+
+### Test 7: Installer Dry Run
+**Spec requirement:** "`--dry-run` prints the plan without changes; stdout carries only the JSON report; mentions transport-resident verification"
+
+**Command:**
+```bash
+node install.js --dir /absolute/path/inst --dry-run
+```
+
+**Output (abbreviated):**
+```json
+{
+  "dryRun": true,
+  "dir": "D:\\...\\.lane\\inst",
+  "manifest": "D:\\...\\.lane\\inst\\manifest.json",
+  "steps": [
+    "create venv...",
+    "...pip install torch...",
+    "...pip install cua_s1...",
+    "download ...safetensors...",
+    "download ...json...",
+    "write manifest.json",
+    "verify provenance (pinned revision), start the resident server, one warm CPU inference, stop it"
+  ],
+  "env": {
+    "HARNESS_SYSTEM_ONE_MODE": "shadow",
+    "HARNESS_SYSTEM_ONE_CONFIG": "..."
+  }
+}
+```
+
+**Validation:**
+- Valid JSON ✓
+- dryRun: true ✓
+- Mentions "resident server" and "verify provenance" ✓
+- stdout is JSON only (no stderr mixed in) ✓
+- No files/directories created in file system ✓
+
+### Test 8: Warm Latency
+**Spec requirement:** "time 30 sequential provider.score calls; report p50/p95"
+
+**Test methodology:**
+- Ensure server is fully ready (6s+ after start)
+- Run 30 sequential score() calls on same request
+- Record time for each call
+- Calculate percentiles
+
+**Results (milliseconds):**
+```
+Call latencies (sorted): [36, 41, 41, 42, 42, 43, 43, 43, 45, 46, 46, 46, 49, 49, 49, 52, 59, 69, 74, 76, 77, 78, 84, 88, 89, 90, 93, 98]
+p50 (median):  49ms
+p95:           93ms
+Min:           36ms
+Max:           98ms
+Average:       59.43ms
+```
+
+**Analysis:**
+- All calls completed within reasonable time
+- p50 under 50ms indicates good typical performance ✓
+- p95 under 100ms acceptable for tier routing (not latency-critical path) ✓
+- Consistent performance across 30 calls ✓
+
+---
+
+## Summary
+
+**Total Spec Clauses Tested:** 24  
+**Passed:** 24  
+**Failed:** 0  
+**Not Run:** 0  
+
+All resident provider (Phase 4 prerequisite) functionality works as specified. The implementation correctly:
+
+1. ✓ Returns `provider-starting` immediately on cold start (no blocking)
+2. ✓ Creates and manages lock/.starting and state/.json files with correct digest-based naming
+3. ✓ State file has exact schema (schemaVersion, pid, port, token:64hex, startedAt)
+4. ✓ Shadow mode emits SYSTEM ONE diagnostic without leaking tokens or prompts
+5. ✓ Server exits automatically after idle timeout and cleans up state
+6. ✓ Detects hard-killed servers and recovers transparently
+7. ✓ CLI stop command works without revealing sensitive data
+8. ✓ Installer dry-run outputs valid JSON describing transport-resident setup
+9. ✓ Warm latency p50=49ms, p95=93ms on real CPU model (CUA-S1-FORMS)
+
+**Evidence Layer:** Live host (real Python server, CPU checkpoint, socket protocol, file I/O) — not mechanism tests or stubs.

package/benchmarks/results/system-one/pre-merge-review-2026-09-23/luna-adversarial.md

@@ -0,0 +1,212 @@
+## Findings
+
+### F-01 — High — Resident token is readable by other Windows users
+
+Reproduction:
+
+```bash
+node harness-everything/scripts/system-one/resident.js start \
+  "<lane-worktree>/.lane/manifest.json"
+
+state=$(node - <<'NODE'
+const p='.lane/manifest.json', fs=require('fs');
+const r=require('./harness-everything/scripts/system-one/resident');
+const m=require('./harness-everything/scripts/system-one/provider').readManifest(p);
+console.log(r.stateFiles(m,p).state)
+NODE
+)
+icacls "$state"
+```
+
+Expected, per spec:
+
+> “State ... is created with mode 0600 (on Windows it inherits the user-profile ACL).”
+
+Observed:
+
+```text
+NT AUTHORITY\Authenticated Users:(I)(M)
+BUILTIN\Users:(I)(RX)
+BUILTIN\Administrators:(I)(F)
+NT AUTHORITY\SYSTEM:(I)(F)
+```
+
+The token is readable by all local Users and modifiable by Authenticated Users in this worktree. The token permits authenticated score, ping, and shutdown operations.
+
+Suggested fix: explicitly apply an owner-only Windows DACL to state, lock, and temporary files, or reject manifests whose parent ACL is not private. POSIX mode `0600` is not sufficient on Windows.
+
+---
+
+### F-02 — Medium — One unauthenticated slow connection blocks the resident server
+
+Reproduction:
+
+```bash
+node harness-everything/scripts/system-one/resident.js start \
+  "<lane-worktree>/.lane/manifest.json"
+
+node .lane/bridge-slowloris.js \
+  "<lane-worktree>/.lane/manifest.json"
+
+node .lane/oversize-slow.js \
+  "<lane-worktree>/.lane/manifest.json"
+```
+
+Expected, per spec:
+
+> “A score never blocks on model loading.”
+
+> “The client ... waits with Atomics.wait (1000 ms deadline).”
+
+Observed:
+
+```json
+{"elapsedMs":1252,"status":0,"stdout":"{\"status\":\"unavailable\",\"reason\":\"provider-timeout\"}","stderr":""}
+{"elapsedMs":1159,"status":0,"stdout":"{\"status\":\"unavailable\",\"reason\":\"provider-timeout\"}","stderr":""}
+```
+
+A connection that sends nothing, or sends an oversized request without closing, occupies the single Python request loop for approximately two seconds. Concurrent legitimate scoring therefore misses the Node client’s one-second deadline.
+
+A 32-connection burst also exceeded the server backlog:
+
+```json
+{"n":24,"errors":7,"unauthorized":17}
+{"n":32,"errors":15,"unauthorized":17}
+```
+
+Suggested fix: enforce a read deadline no greater than the client deadline, avoid blocking drain reads, and process connections concurrently or with an asynchronous event loop. Increase backlog only as a secondary mitigation.
+
+---
+
+### F-03 — Medium — Corrupt state file causes duplicate resident servers
+
+Reproduction:
+
+```bash
+node harness-everything/scripts/system-one/resident.js start \
+  "<lane-worktree>/.lane/manifest.json"
+
+node .lane/corrupt-state.js \
+  "<lane-worktree>/.lane/manifest.json"
+```
+
+Expected, per spec:
+
+> “If no live server exists, the Node provider creates a start lock ... [and] spawns ...”
+
+> “The server ... [deletes] only its own state file.”
+
+Observed:
+
+```json
+{
+  "oldPid":15228,
+  "oldPort":54813,
+  "immediate":{"status":"unavailable","reason":"provider-starting"},
+  "ready":true,
+  "newPid":40732,
+  "newPort":54854,
+  "oldPidAlive":true,
+  "oldPing":{"reply":{"ok":true,"pid":15228}}
+}
+```
+
+The invalid state caused a second Python server to start while the original server remained live and responsive. This duplicates model memory and CPU usage and leaves an orphan that normal `resident stop` cannot see.
+
+Suggested fix: use a durable per-manifest ownership/heartbeat record, quarantine invalid state under an exclusive lock, and prevent a new server from starting until an existing instance is positively ruled out.
+
+---
+
+### F-04 — Medium — Failed executable spawn leaves a non-failed 60-second lock
+
+Reproduction:
+
+```bash
+node harness-everything/scripts/system-one/resident.js stop \
+  "<lane-worktree>/.lane/manifest.json"
+
+node .lane/spawn-failure.js \
+  "<lane-worktree>/.lane/manifest.json"
+```
+
+Expected, per spec:
+
+> “A startup failure writes a bounded reason into the lock and exits; callers report `provider-unavailable` without respawning until the lock is 60 s old.”
+
+Observed:
+
+```json
+{
+  "first":{"status":"unavailable","reason":"provider-starting"},
+  "lock":"{\"createdAt\":1790137422179}",
+  "second":{"status":"unavailable","reason":"provider-starting"},
+  "ensureReady1200ms":false,
+  "stateExists":false,
+  "lockExists":true
+}
+```
+
+When `spawn()` emits its asynchronous `error` event, the lock remains with only `createdAt`; no bounded failure reason is written.
+
+Suggested fix: handle the child `error` event by atomically writing a bounded failure reason to the same lock, and remove or age it according to the normal retry policy.
+
+---
+
+### F-05 — Low — Adapter accepts requests outside the Phase 0 contract
+
+Reproduction:
+
+```bash
+node harness-everything/scripts/system-one/resident.js start \
+  "<lane-worktree>/.lane/manifest.json"
+
+node .lane/protocol-257.js \
+  "<lane-worktree>/.lane/manifest.json"
+```
+
+Expected, per spec:
+
+> “... for `score`, the Phase 0 request.”
+
+The Phase 0 contract limits options to 256 and validates hashes, IDs, schema, and task fields.
+
+Observed:
+
+```json
+{"ok":true,"scoreCount":257,"catalogHashMatches":true}
+```
+
+The direct resident protocol accepted 257 options and returned a successful response despite the request hash no longer matching the mutated option catalog. The normal Node client rejects this before sending, but any process that obtains the token can bypass that validation.
+
+Suggested fix: make `_valid_score_request` enforce the complete Phase 0 request contract, including exact fields, hashes, IDs, and the 256-option maximum.
+
+## Probe results
+
+| Probe | Result |
+|---|---|
+| Wrong/missing token | PASS — bounded `unauthorized` |
+| Non-JSON request | PASS — bounded `invalid-request` |
+| NaN/huge numeric extras | PASS — no crash or echo |
+| >2 MiB request with EOF | PASS — bounded `request-limit` |
+| Exact 224-byte UTF-8 context | PASS — scored |
+| 225-byte context | PASS — bounded `input-too-long` |
+| 256 options | PASS — scored |
+| Slowloris / oversized no-EOF | FAIL — see F-02 |
+| Rapid connection burst | FAIL — backlog refusals, see F-02 |
+| Malformed bridge reply | PASS — `error:"json"` |
+| >1 MiB bridge reply | PASS — `error:"limit"` |
+| 1-second bridge timeout | PASS — observed ~1011 ms |
+| Six-way cold-start race | PASS — one isolated state/Python server observed |
+| Hard-kill recovery | PASS — recovery worked after `Stop-Process -Force`; direct `taskkill.exe` was blocked by shell/sandbox behavior |
+| Corrupt state | FAIL — duplicate server, see F-03 |
+| Live unrelated port | Observed `provider-json` with state retained; consistent with malformed-reply mapping, not counted separately |
+| Stale lock older than 60 s | PASS — replaced and restarted |
+| Spawn failure | FAIL — see F-04 |
+| Shadow secret-output check | PASS — token, checkpoint path, manifest path, and traceback absent |
+| Hermetic tests with shadow config | PASS — 12/12 and router workflow 0 failures |
+| Hermetic tests with variables unset | PASS — 12/12 and router workflow 0 failures |
+| Existing resident tests | PASS — 7/7 |
+| Existing adapter tests | PASS — 8/8 |
+| Final cleanup | PASS — no state/lock file; all tester-started PIDs dead |
+
+No product code or tests were modified. Only `.lane/` probe scripts, the existing manifest, and the local todo file are untracked.

package/benchmarks/results/system-one/pre-merge-review-2026-09-23/skill-top1-probe.js

@@ -0,0 +1,83 @@
+'use strict';
+// Diagnostic: does the System One scorer's top-1 point at the right skill for historical routing inputs?
+// Usage: node skill-probe.js <repoRoot> <absolute manifest> <out.json>
+const fs = require('node:fs');
+const path = require('node:path');
+const [root, manifestPath, outPath] = process.argv.slice(2);
+const S1 = path.join(root, 'harness-everything/scripts/system-one');
+const { createRequest, decide } = require(path.join(S1, 'contract'));
+const provider = require(path.join(S1, 'provider'));
+const resident = require(path.join(S1, 'resident'));
+const { run: route } = require(path.join(root, 'harness-everything/scripts/tier-router'));
+
+const OPTION_BYTES = 96;
+const CONTEXT_BYTES = 224;
+const bytes = s => Buffer.byteLength(s, 'utf8');
+function fit(text, max) { // explicit catalog construction: cut at a word boundary, never mid-character
+  if (bytes(text) <= max) return text;
+  const words = text.split(/\s+/); let out = '';
+  for (const w of words) { const next = out ? `${out} ${w}` : w; if (bytes(next) > max) break; out = next; }
+  return out;
+}
+const skills = fs.readdirSync(path.join(root, 'evals')).filter(d => fs.existsSync(path.join(root, d, 'SKILL.md'))).sort();
+const options = skills.map(id => {
+  const md = fs.readFileSync(path.join(root, id, 'SKILL.md'), 'utf8');
+  const desc = (md.match(/^description:\s*"?(.*?)"?\s*$/m) || [])[1] || id;
+  return { id, text: fit(`${id}: ${desc.replace(/\\"/g, '"')}`, OPTION_BYTES) };
+});
+
+const cfg = require(path.join(root, 'harness-everything/scripts/routing-keywords.json'));
+const keywordGold = new Map();
+for (const g of cfg.guideGroups) {
+  const gs = [...new Set((g.guides || []).map(s => (s.match(/^- ([^/\s]+)\//) || [])[1]).filter(id => skills.includes(id)))];
+  for (const kw of g.keywords || []) {
+    const set = keywordGold.get(kw) || new Set(); gs.forEach(s => set.add(s)); keywordGold.set(kw, set);
+  }
+}
+const setA = [...keywordGold].map(([kw, gold]) => ({ set: 'A-keyword', context: kw, gold: [...gold] }));
+const setB = [];
+for (const id of skills) {
+  const dir = path.join(root, 'evals', id, 'tasks');
+  if (!fs.existsSync(dir)) continue;
+  for (const f of fs.readdirSync(dir).filter(x => x.endsWith('.yaml'))) {
+    const y = fs.readFileSync(path.join(dir, f), 'utf8');
+    if (!/should_trigger:\s*true/.test(y)) continue;
+    const m = y.match(/^\s*prompt:\s*(?:"((?:[^"\\]|\\.)*)"|'([^']*)'|(.+))\s*$/m);
+    if (m) setB.push({ set: 'B-eval-prompt', context: JSON.parse(`"${(m[1] ?? m[2] ?? m[3]).replace(/\n/g, ' ')}"`.replace(/^""$/, '""')), gold: [id], task: f });
+  }
+}
+
+const manifest = provider.readManifest(manifestPath);
+if (manifest.transport === 'resident' && !resident.ensureReady(manifest, manifestPath, 60000)) throw new Error('resident not ready');
+const originalLog = console.log;
+const rows = [];
+for (const item of [...setA, ...setB]) {
+  const row = { ...item, bytes: bytes(item.context) };
+  if (row.bytes > CONTEXT_BYTES) { rows.push({ ...row, status: 'context-over-limit' }); continue; }
+  const request = createRequest('skill', item.context, options);
+  const result = provider.score(request, manifestPath);
+  if (result.status !== 'scored') { rows.push({ ...row, status: result.reason }); continue; }
+  const d = decide(request, result.response);
+  const ranked = [...result.response.scores].sort((a, b) => b.probability - a.probability);
+  let lexical = null;
+  if (item.set === 'B-eval-prompt') { // lexical baseline: does the keyword router recommend the gold skill at all?
+    console.log = () => {};
+    try { const out = []; console.log = (...a) => out.push(a.join(' ')); route(item.context); lexical = out.join('\n').includes(`- ${item.gold[0]}/`); }
+    finally { console.log = originalLog; }
+  }
+  rows.push({ ...row, status: 'scored', decision: d.reason, top1: ranked[0].id, top1p: ranked[0].probability,
+    top3: ranked.slice(0, 3).map(r => r.id), correctTop1: item.gold.includes(ranked[0].id),
+    correctTop3: ranked.slice(0, 3).some(r => item.gold.includes(r.id)), lexicalRecommendsGold: lexical });
+}
+const summarize = set => {
+  const r = rows.filter(x => x.set === set); const s = r.filter(x => x.status === 'scored');
+  const dist = {}; s.forEach(x => { dist[x.top1] = (dist[x.top1] || 0) + 1; });
+  return { total: r.length, scored: s.length, rejected: r.length - s.length,
+    top1Correct: s.filter(x => x.correctTop1).length, top3Correct: s.filter(x => x.correctTop3).length,
+    highConfidenceWrong: s.filter(x => !x.correctTop1 && x.top1p >= 0.9).length,
+    decisions: [...new Set(s.map(x => x.decision))], top1Distribution: Object.entries(dist).sort((a, b) => b[1] - a[1]),
+    lexicalRecommendsGold: set === 'B-eval-prompt' ? s.filter(x => x.lexicalRecommendsGold).length : undefined };
+};
+const report = { options, summary: { A: summarize('A-keyword'), B: summarize('B-eval-prompt') }, rows };
+fs.writeFileSync(outPath, JSON.stringify(report, null, 2));
+console.log(JSON.stringify(report.summary, null, 1));