harness-everything 0.23.0 → 0.25.0

package/docs/system-one-training.md

@@ -0,0 +1,131 @@
+# System One training data and weights
+
+This document defines how the `harness-routing-v1` tier checkpoint for
+[System One](system-one-routing.md) is trained, calibrated and released
+(issue [#233](https://github.com/dyphn1/Harness-everything/issues/233), Phase 4).
+Gold labels follow [system-one-corpus.md](system-one-corpus.md). The owner-reviewed
+holdout in that document is used only for evaluation. It is never used for training,
+threshold selection or labeler tuning.
+
+## Pipeline
+
+1. **Collect**: copy the owner's own prompts from local assistant histories into a local
+   dataset (`scripts/system-one-collect.js`).
+2. **Label**: an LLM labels each prompt against the owner's rules. The labeler is checked
+   once against the holdout, and the owner spot-checks a sample
+   (`scripts/system-one-label.js`).
+3. **Train**: train a fresh `cua_s1` `tinyx` scorer on the train split and select the epoch
+   on the validation split.
+4. **Calibrate**: choose the confidence and margin thresholds on validation.
+5. **Evaluate**: produce one paired report on the holdout.
+6. **Release**: publish the weights as a GitHub release asset. The installer pins them by
+   SHA-256.
+
+## Sources
+
+Only text the owner typed is collected. Injected context, tool output and assistant text are
+never collected.
+
+| Source | Location | User text |
+| --- | --- | --- |
+| Claude Code | `~/.claude/projects/*/*.jsonl` | `type: "user"` records that are not meta or sidechain, string content or text parts |
+| Codex | `~/.codex/sessions/**/*.jsonl` | `event_msg` `user_message.message`. Older sessions without it fall back to `response_item` user text |
+| Copilot CLI | `~/.copilot/session-state/*/events.jsonl` | `user.message` `data.content` |
+| VS Code Copilot Chat | `%APPDATA%/Code/User/workspaceStorage/*/chatSessions/*.json(l)` and `globalStorage/emptyWindowChatSessions` | `requests[].message.text` in snapshots (`kind: 0`) and request pushes (`kind: 2`, key `["requests"]`) |
+
+The following are dropped:
+
+- Text that starts with an injected wrapper tag, such as `<environment_context>`,
+  `<system-reminder>`, `<command-…>`, `<task-notification>` or `<ide_…>`.
+- Tool results.
+- Slash commands.
+- Session-continuation summaries ("This session is being continued…").
+- Empty text.
+
+Codex sessions are kept only when they are interactive, meaning `session_meta` has a
+`source` of `vscode` or `cli` and an originator that is neither `Claude Code` nor
+`codex_exec`. Exec sessions, sub-agent threads and sessions another agent started contain
+prompts written by an agent, not by the owner. VS Code terminal notifications
+(`[Terminal … notification …`) are dropped as well.
+
+## Filters, families and splits
+
+- **Length**: at most 1024 UTF-8 bytes, which matches the checkpoint's `context_tokens`. A
+  longer prompt is still refused at inference and falls back to lexical routing. It is never
+  truncated.
+- **Deduplication**: exact duplicates after whitespace normalization are dropped. The first
+  occurrence is kept.
+- **Holdout leakage**: a prompt is dropped if its character-trigram Jaccard similarity to any
+  holdout prompt is at least 0.5. The same applies if either prompt contains the other after
+  normalization and the shorter one is more than 12 characters long. This matters because
+  the holdout includes rewrites of local prompt shapes.
+- **Family**: the source session. Prompts from one conversation never span splits.
+- **Split**: 15% of families go to `validation`, chosen by a stable SHA-256 of the family ID.
+  The rest go to `train`.
+
+## Privacy
+
+- Collected prompts and labels stay on the local machine under
+  `~/.agents/harness-everything/system-one/training/` and are **never committed**. The
+  repository holds only the scripts, aggregate statistics and the SHA-256 of each dataset
+  file.
+- Email addresses and user home paths are replaced with placeholders before storage. Names
+  and product terms are not reliably detectable and remain.
+- The released weights come from a 4-way byte-level classifier with about 0.7M parameters. It
+  has no text-generation head, but a trained model can still encode fragments of its data.
+  The owner accepted this for their own prompts.
+- Labeling sends the prompts to the Claude API. The owner authorized this.
+
+## Labeling
+
+- **Labeler**: `claude -p` with Sonnet.
+  - Flags: `--setting-sources ""` (no hooks or plugins), `--no-session-persistence` (labeling
+    sessions never become collectable history), `--tools ""` and a JSON schema.
+  - The instructions contain the gold table and rules from
+    [system-one-corpus.md](system-one-corpus.md).
+  - Prompts go in batches of 100. `--max-batches N` stops after N batches so a
+    memory-constrained host can label in short foreground runs that resume from the
+    labels already written. Every returned index must be labeled exactly once with
+    `tier1`, `tier2`, `tier3` or `null`. A malformed batch is retried once and then
+    recorded as failed. It is never partially accepted.
+- **Labeler check**: the labeler labels the 215 holdout prompts once, before any training
+  data is labeled. The report gives agreement with the owner's gold and the per-class recall.
+  - Labeling proceeds only when agreement is at least 80%.
+  - The instructions are never tuned on the holdout. If the check fails, the owner reviews
+    the disagreements and decides how to proceed.
+- **Owner spot-check**: a deterministic sample of the labeled training data is reviewed on
+  the review page, and the disagreement rate is reported.
+  - Each case shows the preceding prompts from the same session as read-only context.
+    Without it, short replies read as confirmations: the first spot-check (103 cases,
+    no context) disagreed 51% and was not used to override labels.
+  - A checkpoint may be trained before the spot-check, but it is released only when
+    disagreement is below 20%.
+  - Decisions made with context replace the LLM labels for those cases.
+
+## Training and calibration
+
+- **Model**: a new `cua_s1` `tinyx` scorer built with `make_system`.
+  - Config: `width` 128, `rank` 128, `layers` 2, `heads` 4, `context_tokens` 1024 and
+    `option_tokens` 96.
+  - Weights are initialized fresh, not from the forms checkpoint, whose domain and
+    positional range differ.
+- **Target**: each example is the fixed tier catalog with the index of the gold option.
+  `null` maps to `unclassified`.
+- **Optimization**:
+  - Cross-entropy, AdamW, a fixed seed and CPU.
+  - The epoch with the lowest validation negative log-likelihood is kept.
+  - The checkpoint metadata records the dataset hashes, the config and the validation metrics.
+- **Calibration**: `minConfidence` and `minMargin` are chosen on validation to maximize
+  coverage, subject to an accepted precision of at least 98%. The router currently uses the
+  Phase 0 defaults (0.9 and 0.2). Carrying calibrated thresholds into the manifest is a
+  separate contract change.
+
+## Evaluation and release
+
+- **Evaluation**: `scripts/evaluate-system-one.js` runs once on
+  `benchmarks/fixtures/system-one-holdout.json` with the new manifest. The rollout gates and
+  the lexical baseline (macro-F1 0.254) decide the result. The report goes under
+  `benchmarks/results/system-one/`.
+- **Release**: the `.safetensors` and `.json` files are attached to a GitHub release. The
+  installer's pins gain the new model ID, revision, `domain: "harness-routing-v1"` and both
+  SHA-256 values. Until every rollout gate passes, the router default stays `off`.

package/harness-everything/SKILL.md

@@ -4,7 +4,7 @@ description: "Route software/project work through Harness: classify tier/topolog
 license: Apache-2.0
 metadata:
   author: Miya Daniel
-  version: 0.22.1
+  version: 0.24.0
 ---
 
 # Harness Everything

package/harness-everything/scripts/system-one/provider.js

@@ -10,10 +10,17 @@ const unavailable = reason => ({ status: 'unavailable', reason });
 const exact = (value, keys) => value && typeof value === 'object' && !Array.isArray(value)
   && Object.keys(value).length === keys.length && keys.every(k => Object.hasOwn(value, k));
 const label = value => typeof value === 'string' && value.length > 0 && value.length <= 128 && !/[\u0000-\u001f]/.test(value);
+// Thresholds calibrated on validation for this checkpoint (docs/system-one-training.md).
+const inRange = (v, lo, hi) => typeof v === 'number' && Number.isFinite(v) && v >= lo && v <= hi;
+const validAcceptance = a => exact(a, ['minConfidence', 'minMargin']) && inRange(a.minConfidence, 0.5, 0.99) && inRange(a.minMargin, 0, 0.9);
+// A scored result carries the manifest's thresholds so every caller decides with them.
+const withAcceptance = (result, manifest) => (result && result.status === 'scored' && manifest.acceptance
+  ? { ...result, acceptance: { ...manifest.acceptance } } : result);
 function validateManifest(m) {
   const required = ['schemaVersion', 'python', 'checkpoint', 'weightsSha256', 'configSha256', 'modelId', 'revision', 'domain'];
   if (!m || typeof m !== 'object' || Array.isArray(m) || required.some(k => !Object.hasOwn(m, k))
-    || Object.keys(m).some(k => ![...required, 'timeoutMs', 'transport', 'idleTimeoutMs'].includes(k)) || m.schemaVersion !== 1
+    || Object.keys(m).some(k => ![...required, 'timeoutMs', 'transport', 'idleTimeoutMs', 'acceptance'].includes(k)) || m.schemaVersion !== 1
+    || (m.acceptance !== undefined && !validAcceptance(m.acceptance))
     || ['python', 'modelId', 'revision', 'domain'].some(k => typeof m[k] !== 'string' || !m[k].trim() || m[k].length > 4096 || m[k].includes('\0'))
     || typeof m.checkpoint !== 'string' || !path.isAbsolute(m.checkpoint) || path.extname(m.checkpoint) !== '.safetensors'
     || ['weightsSha256', 'configSha256'].some(k => typeof m[k] !== 'string' || !/^[a-f0-9]{64}$/.test(m[k]))
@@ -50,8 +57,8 @@ function score(request, manifestPath) {
   validateRequest(request);
   let manifest;
   try { manifest = readManifest(manifestPath); } catch (_) { return unavailable('provider-config'); }
-  if (manifest.transport === 'resident') return require('./resident').score({ ...request }, manifest, manifestPath);
-  return oneShot(request, manifest);
+  if (manifest.transport === 'resident') return withAcceptance(require('./resident').score({ ...request }, manifest, manifestPath), manifest);
+  return withAcceptance(oneShot(request, manifest), manifest);
 }
 // Async callers (the hook entry) reach a resident server without the sync worker bridge.
 // One-shot has no server to overlap with, so it runs the same child synchronously.
@@ -59,8 +66,8 @@ async function scoreAsync(request, manifestPath) {
   validateRequest(request);
   let manifest;
   try { manifest = readManifest(manifestPath); } catch (_) { return unavailable('provider-config'); }
-  if (manifest.transport === 'resident') return require('./resident').scoreAsync({ ...request }, manifest, manifestPath);
-  return oneShot(request, manifest);
+  if (manifest.transport === 'resident') return withAcceptance(await require('./resident').scoreAsync({ ...request }, manifest, manifestPath), manifest);
+  return withAcceptance(oneShot(request, manifest), manifest);
 }
 // Send the exact validated manifest to the child on stdin: no config reread race.
 function oneShot(request, manifest) {

package/harness-everything/scripts/system-one/router.js

@@ -26,7 +26,7 @@ function selectTier(input, env = process.env, scorer = provider.score) {
     const allowed = ['provider-timeout', 'provider-output-limit', 'provider-exit', 'provider-json', 'provider-config', 'provider-unavailable', 'provider-starting'];
     return fallback(allowed.includes(result?.reason) ? result.reason : 'provider-unavailable');
   }
-  const decision = decide(request, result.response);
+  const decision = decide(request, result.response, result.acceptance ? { ...result.acceptance } : {});
   if (decision.selectedId === 'unclassified') Object.assign(decision, { selectedId: null, status: 'abstain', reason: 'unclassified' });
   const accepted = decision.status === 'accepted';
   const belowFloor = accepted && input.floor != null && RANK[decision.selectedId] < RANK[input.floor];

package/opencode-plugin/plugin.json

@@ -1,7 +1,7 @@
 {
   "name": "harness-enforcement",
   "description": "Repository inventory for the opencode enforcement plugin and its standalone helper",
-  "version": "0.23.0",
+  "version": "0.25.0",
   "runtime": "index.mjs",
   "hooks": {
     "verify": "hooks/verify.js"

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "harness-everything",
-  "version": "0.23.0",
+  "version": "0.25.0",
   "description": "Harness OS: An Orchestrated Agent Operating System for AI-driven Development",
   "main": "harness-everything/scripts/bootstrap.js",
   "bin": {

package/plugins/harness-everything/.codex-plugin/plugin.json

@@ -1,6 +1,6 @@
 {
   "name": "harness-everything",
-  "version": "0.23.0",
+  "version": "0.25.0",
   "description": "Invariant-first engineering harness for routing, verification, and failure recovery while preserving agent autonomy.",
   "author": {
     "name": "Miya Daniel",

package/plugins/harness-everything/plugin.json

@@ -1,7 +1,7 @@
 {
   "$schema": "https://agent-plugins.org/schemas/1.0.0/plugin.schema.json",
   "name": "harness-everything",
-  "version": "0.23.0",
+  "version": "0.25.0",
   "description": "Invariant-first engineering harness for routing, verification, and failure recovery while preserving agent autonomy.",
   "author": {
     "name": "Miya Daniel",

package/plugins/harness-everything/skills/harness-everything/SKILL.md

@@ -4,7 +4,7 @@ description: "Route software/project work through Harness: classify tier/topolog
 license: Apache-2.0
 metadata:
   author: Miya Daniel
-  version: 0.22.1
+  version: 0.24.0
 ---
 
 # Harness Everything

package/plugins/harness-everything/skills/harness-everything/scripts/system-one/provider.js

@@ -10,10 +10,17 @@ const unavailable = reason => ({ status: 'unavailable', reason });
 const exact = (value, keys) => value && typeof value === 'object' && !Array.isArray(value)
   && Object.keys(value).length === keys.length && keys.every(k => Object.hasOwn(value, k));
 const label = value => typeof value === 'string' && value.length > 0 && value.length <= 128 && !/[\u0000-\u001f]/.test(value);
+// Thresholds calibrated on validation for this checkpoint (docs/system-one-training.md).
+const inRange = (v, lo, hi) => typeof v === 'number' && Number.isFinite(v) && v >= lo && v <= hi;
+const validAcceptance = a => exact(a, ['minConfidence', 'minMargin']) && inRange(a.minConfidence, 0.5, 0.99) && inRange(a.minMargin, 0, 0.9);
+// A scored result carries the manifest's thresholds so every caller decides with them.
+const withAcceptance = (result, manifest) => (result && result.status === 'scored' && manifest.acceptance
+  ? { ...result, acceptance: { ...manifest.acceptance } } : result);
 function validateManifest(m) {
   const required = ['schemaVersion', 'python', 'checkpoint', 'weightsSha256', 'configSha256', 'modelId', 'revision', 'domain'];
   if (!m || typeof m !== 'object' || Array.isArray(m) || required.some(k => !Object.hasOwn(m, k))
-    || Object.keys(m).some(k => ![...required, 'timeoutMs', 'transport', 'idleTimeoutMs'].includes(k)) || m.schemaVersion !== 1
+    || Object.keys(m).some(k => ![...required, 'timeoutMs', 'transport', 'idleTimeoutMs', 'acceptance'].includes(k)) || m.schemaVersion !== 1
+    || (m.acceptance !== undefined && !validAcceptance(m.acceptance))
     || ['python', 'modelId', 'revision', 'domain'].some(k => typeof m[k] !== 'string' || !m[k].trim() || m[k].length > 4096 || m[k].includes('\0'))
     || typeof m.checkpoint !== 'string' || !path.isAbsolute(m.checkpoint) || path.extname(m.checkpoint) !== '.safetensors'
     || ['weightsSha256', 'configSha256'].some(k => typeof m[k] !== 'string' || !/^[a-f0-9]{64}$/.test(m[k]))
@@ -50,8 +57,8 @@ function score(request, manifestPath) {
   validateRequest(request);
   let manifest;
   try { manifest = readManifest(manifestPath); } catch (_) { return unavailable('provider-config'); }
-  if (manifest.transport === 'resident') return require('./resident').score({ ...request }, manifest, manifestPath);
-  return oneShot(request, manifest);
+  if (manifest.transport === 'resident') return withAcceptance(require('./resident').score({ ...request }, manifest, manifestPath), manifest);
+  return withAcceptance(oneShot(request, manifest), manifest);
 }
 // Async callers (the hook entry) reach a resident server without the sync worker bridge.
 // One-shot has no server to overlap with, so it runs the same child synchronously.
@@ -59,8 +66,8 @@ async function scoreAsync(request, manifestPath) {
   validateRequest(request);
   let manifest;
   try { manifest = readManifest(manifestPath); } catch (_) { return unavailable('provider-config'); }
-  if (manifest.transport === 'resident') return require('./resident').scoreAsync({ ...request }, manifest, manifestPath);
-  return oneShot(request, manifest);
+  if (manifest.transport === 'resident') return withAcceptance(await require('./resident').scoreAsync({ ...request }, manifest, manifestPath), manifest);
+  return withAcceptance(oneShot(request, manifest), manifest);
 }
 // Send the exact validated manifest to the child on stdin: no config reread race.
 function oneShot(request, manifest) {

package/plugins/harness-everything/skills/harness-everything/scripts/system-one/router.js

@@ -26,7 +26,7 @@ function selectTier(input, env = process.env, scorer = provider.score) {
     const allowed = ['provider-timeout', 'provider-output-limit', 'provider-exit', 'provider-json', 'provider-config', 'provider-unavailable', 'provider-starting'];
     return fallback(allowed.includes(result?.reason) ? result.reason : 'provider-unavailable');
   }
-  const decision = decide(request, result.response);
+  const decision = decide(request, result.response, result.acceptance ? { ...result.acceptance } : {});
   if (decision.selectedId === 'unclassified') Object.assign(decision, { selectedId: null, status: 'abstain', reason: 'unclassified' });
   const accepted = decision.status === 'accepted';
   const belowFloor = accepted && input.floor != null && RANK[decision.selectedId] < RANK[input.floor];

package/sbom.cdx.json

@@ -2,23 +2,23 @@
   "$schema": "https://cyclonedx.org/schema/bom-1.7.schema.json",
   "bomFormat": "CycloneDX",
   "specVersion": "1.7",
-  "serialNumber": "urn:uuid:a1c20a4d-4ba9-561e-92d1-aae9c14cdc08",
+  "serialNumber": "urn:uuid:a66915ec-6b2f-51ca-a5d4-4a642ce3a914",
   "version": 1,
   "metadata": {
-    "timestamp": "2026-09-23T11:44:30.450Z",
+    "timestamp": "2026-09-23T15:52:59.593Z",
     "tools": [
       {
         "vendor": "Harness Everything",
         "name": "generate-sbom.js",
-        "version": "0.23.0"
+        "version": "0.25.0"
       }
     ],
     "component": {
       "type": "application",
-      "bom-ref": "pkg:npm/harness-everything@0.23.0",
+      "bom-ref": "pkg:npm/harness-everything@0.25.0",
       "name": "harness-everything",
-      "version": "0.23.0",
-      "purl": "pkg:npm/harness-everything@0.23.0"
+      "version": "0.25.0",
+      "purl": "pkg:npm/harness-everything@0.25.0"
     }
   },
   "components": [
@@ -1999,7 +1999,7 @@
       ]
     },
     {
-      "ref": "pkg:npm/harness-everything@0.23.0",
+      "ref": "pkg:npm/harness-everything@0.25.0",
       "dependsOn": [
         "pkg:npm/%40commitlint/cli@19.8.1",
         "pkg:npm/%40commitlint/config-conventional@19.8.1",

package/scripts/evaluate-system-one.js

@@ -29,7 +29,7 @@ async function run(corpus, manifest) {
       for (let i = 0; i < 2; i++) {
         const start = performance.now();
         const result = await scoreAsync(c.request, manifest);
-        const decision = result.status === 'scored' ? decide(c.request, result.response) : result;
+        const decision = result.status === 'scored' ? decide(c.request, result.response, result.acceptance ? { ...result.acceptance } : {}) : result;
         if (decision.selectedId === 'unclassified') Object.assign(decision, { status: 'abstain', reason: 'unclassified', selectedId: null });
         const scores = ['accepted', 'abstain'].includes(decision.status) ? result.response.scores.map(s => s.probability) : null;
         runs.push({ decision, scores, latencyMs: performance.now() - start, coldStart: !warm });

package/scripts/system-one-calibrate.js

@@ -0,0 +1,44 @@
+#!/usr/bin/env node
+'use strict';
+// Chooses System One acceptance thresholds on validation scores (docs/system-one-training.md).
+// The holdout is never used here.
+const fs = require('node:fs');
+
+const IDS = ['tier1', 'tier2', 'tier3', 'unclassified'];
+// Same acceptance rule as contract.decide: top probability and top-minus-second margin, unclassified abstains.
+function decideRow(probs, { minConfidence, minMargin }) {
+  const order = probs.map((p, i) => [p, i]).sort((a, b) => b[0] - a[0] || a[1] - b[1]);
+  const [top, second] = order;
+  const margin = top[0] - second[0];
+  if (margin === 0 || top[0] < minConfidence || margin < minMargin || IDS[top[1]] === 'unclassified') return null;
+  return IDS[top[1]];
+}
+function calibrate(rows, { minPrecision = 0.98 } = {}) {
+  let best = null;
+  for (let c = 50; c <= 99; c++) {
+    for (let m = 0; m <= 90; m += 5) {
+      const t = { minConfidence: c / 100, minMargin: m / 100 };
+      let accepted = 0; let correct = 0;
+      for (const r of rows) { const d = decideRow(r.probs, t); if (d) { accepted++; if (d === (r.gold === null ? 'unclassified' : r.gold)) correct++; } }
+      const precision = accepted ? correct / accepted : 0;
+      if (!accepted || precision < minPrecision) continue;
+      const coverage = accepted / rows.length;
+      // Highest coverage wins; ties keep the first (lowest) thresholds scanned.
+      if (!best || coverage > best.coverage) best = { ...t, coverage, precision, accepted };
+    }
+  }
+  return best ? { feasible: true, rows: rows.length, minPrecision, ...best }
+    : { feasible: false, rows: rows.length, minPrecision, minConfidence: null, minMargin: null, coverage: 0, precision: null, accepted: 0 };
+}
+
+if (require.main === module) {
+  try {
+    const [input, output, extra] = process.argv.slice(2);
+    if (!input || !output || extra) throw new Error('usage: system-one-calibrate.js <val-scores.jsonl> <calibration.json>');
+    const rows = fs.readFileSync(input, 'utf8').split('\n').filter(Boolean).map(l => JSON.parse(l));
+    const result = calibrate(rows);
+    fs.writeFileSync(output, `${JSON.stringify(result, null, 2)}\n`);
+    console.log(JSON.stringify(result));
+  } catch (err) { console.error(`system-one-calibrate: ${err.message}`); process.exitCode = 1; }
+}
+module.exports = { decideRow, calibrate };

package/scripts/system-one-collect.js

@@ -0,0 +1,200 @@
+#!/usr/bin/env node
+'use strict';
+// Collects the owner's own prompts from local assistant histories into a local training dataset.
+// See docs/system-one-training.md. The dataset is never written inside the repository.
+const fs = require('node:fs');
+const os = require('node:os');
+const path = require('node:path');
+const { createHash } = require('node:crypto');
+const { StringDecoder } = require('node:string_decoder');
+
+const REPO = path.resolve(__dirname, '..');
+const MAX_BYTES = 1024;
+const VALIDATION_PERCENT = 15;
+// Whole-text wrappers injected by hosts or hooks; text that starts with one is not typed by the owner.
+const WRAPPER = /^(<(environment_context|user_instructions|recommended_plugins|send_user_message_question_reply|command-|local-command|task-notification|bash-|turn_aborted|user_shell_command|skill)|\[Terminal [0-9a-f-]+ notification)/i;
+// Codex sessions typed by the owner: interactive IDE/desktop/terminal ones, not exec lanes, sub-agents or agent-started sessions.
+const interactiveCodex = meta => !!meta && ['vscode', 'cli'].includes(meta.source) && !['Claude Code', 'codex_exec'].includes(meta.originator);
+// Blocks a host inserts inside typed text; they are removed and the rest is kept.
+const INSERTED = /<(system-reminder|ide_[a-z_]+)>[\s\S]*?<\/\1>/gi;
+const sha = text => createHash('sha256').update(text, 'utf8').digest('hex');
+
+function* readLines(file) {
+  let fd;
+  try { fd = fs.openSync(file, 'r'); } catch (_) { return; }
+  const decoder = new StringDecoder('utf8'); const buf = Buffer.alloc(4 * 1024 * 1024); let rest = '';
+  try {
+    for (let n; (n = fs.readSync(fd, buf, 0, buf.length, null)) > 0;) {
+      const parts = (rest + decoder.write(buf.subarray(0, n))).split('\n');
+      rest = parts.pop();
+      yield* parts;
+    }
+    rest += decoder.end();
+    if (rest) yield rest;
+  } finally { fs.closeSync(fd); }
+}
+const parse = line => { try { return JSON.parse(line); } catch (_) { return null; } };
+function walk(dir, pred, out = [], depth = 0) {
+  if (depth > 8) return out;
+  let entries = [];
+  try { entries = fs.readdirSync(dir, { withFileTypes: true }).sort((a, b) => a.name.localeCompare(b.name)); } catch (_) { return out; }
+  for (const e of entries) {
+    const p = path.join(dir, e.name);
+    if (e.isDirectory()) walk(p, pred, out, depth + 1); else if (pred(p)) out.push(p);
+  }
+  return out;
+}
+// Typed text after removing inserted blocks; null when the text is a wrapper or nothing is left.
+function typed(text) {
+  if (typeof text !== 'string') return null;
+  const t = text.replace(INSERTED, '').trim();
+  return !t || WRAPPER.test(t) ? null : t;
+}
+
+function fromClaude(file) {
+  const rows = [];
+  for (const line of readLines(file)) {
+    const e = parse(line);
+    if (!e || e.type !== 'user' || e.isMeta || e.isSidechain) continue;
+    const c = e.message && e.message.content;
+    const raw = typeof c === 'string' ? c : Array.isArray(c) ? c.filter(p => p && p.type === 'text').map(p => p.text).join('\n') : null;
+    const t = typed(raw); if (t) rows.push(t);
+  }
+  return rows;
+}
+function fromCodex(file) {
+  const events = []; const items = []; let meta = null;
+  for (const line of readLines(file)) {
+    const e = parse(line); if (!e || !e.payload) continue;
+    if (e.type === 'session_meta' && !meta) { meta = e.payload; if (!interactiveCodex(meta)) return []; continue; }
+    if (e.type === 'event_msg' && e.payload.type === 'user_message') { const t = typed(e.payload.message); if (t) events.push(t); }
+    if (e.type === 'response_item' && e.payload.role === 'user') for (const c of e.payload.content || []) { const t = typed(c && c.text); if (t) items.push(t); }
+  }
+  if (!interactiveCodex(meta)) return [];
+  // Event messages are the typed text; response items repeat it with injected context, so they are a fallback only.
+  return events.length ? events : items;
+}
+function fromCopilotCli(file) {
+  const rows = [];
+  for (const line of readLines(file)) {
+    const e = parse(line);
+    if (e && e.type === 'user.message' && e.data) { const t = typed(e.data.content); if (t) rows.push(t); }
+  }
+  return rows;
+}
+function fromVscode(file) {
+  const rows = [];
+  const take = requests => { for (const r of Array.isArray(requests) ? requests : [requests]) { const t = typed(r && r.message && r.message.text); if (t) rows.push(t); } };
+  if (file.endsWith('.json')) {
+    let j = null; try { j = JSON.parse(fs.readFileSync(file, 'utf8')); } catch (_) { /* unreadable snapshot */ }
+    if (j) take(j.requests || []);
+    return rows;
+  }
+  for (const line of readLines(file)) {
+    const e = parse(line); if (!e) continue;
+    if (e.kind === 0 && e.v) take(e.v.requests || []);
+    else if (e.kind === 2 && Array.isArray(e.k) && e.k.length === 1 && e.k[0] === 'requests') take(e.v);
+  }
+  return rows;
+}
+
+// Every prompt with its source and a family derived from its session file (no path is stored).
+function readSources({ home, appdata }) {
+  const sources = [
+    ['claude', walk(path.join(home, '.claude', 'projects'), p => p.endsWith('.jsonl')), fromClaude],
+    ['codex', walk(path.join(home, '.codex', 'sessions'), p => p.endsWith('.jsonl')), fromCodex],
+    ['copilot-cli', walk(path.join(home, '.copilot', 'session-state'), p => path.basename(p) === 'events.jsonl'), fromCopilotCli],
+    ['vscode', [
+      ...walk(path.join(appdata, 'Code', 'User', 'workspaceStorage'), p => /[\\/]chatSessions[\\/][^\\/]+\.jsonl?$/.test(p)),
+      ...walk(path.join(appdata, 'Code', 'User', 'globalStorage', 'emptyWindowChatSessions'), p => /\.jsonl?$/.test(p)),
+    ], fromVscode],
+  ];
+  const rows = [];
+  for (const [source, files, read] of sources) {
+    for (const file of files) {
+      const family = `${source}:${sha(path.relative(home, file)).slice(0, 16)}`;
+      for (const text of read(file)) rows.push({ source, family, text });
+    }
+  }
+  return rows;
+}
+
+function keep(text) {
+  const t = typeof text === 'string' ? text.trim() : '';
+  return t.length > 0 && Buffer.byteLength(t, 'utf8') <= MAX_BYTES && !/^\/\S/.test(t) && !t.startsWith('This session is being continued');
+}
+function redact(text) {
+  return text
+    .replace(/[\w.+-]+@[\w-]+\.[\w.-]+/g, '<email>')
+    .replace(/[A-Za-z]:\\Users\\[^\\\s]+/gi, '<home>')
+    .replace(/[A-Za-z]:\/Users\/[^/\s]+/gi, '<home>')
+    .replace(/\/(home|Users)\/[^/\s]+/g, '<home>');
+}
+const squash = text => text.replace(/\s+/g, ' ').trim();
+function dedupe(rows) {
+  const seen = new Set();
+  return rows.filter(r => { const k = squash(r.text); if (seen.has(k)) return false; seen.add(k); return true; });
+}
+const normalize = text => text.toLowerCase().replace(/[\p{P}\p{S}]+/gu, ' ').replace(/\s+/g, ' ').trim();
+function trigrams(text) {
+  const s = new Set();
+  if (text.length < 3) { if (text) s.add(text); return s; }
+  for (let i = 0; i <= text.length - 3; i++) s.add(text.slice(i, i + 3));
+  return s;
+}
+function jaccard(a, b) {
+  const x = trigrams(a); const y = trigrams(b);
+  if (!x.size && !y.size) return 1;
+  let inter = 0; for (const g of x) if (y.has(g)) inter++;
+  return inter / (x.size + y.size - inter);
+}
+// True when a prompt near-duplicates any holdout prompt (evaluation data must not leak into training).
+function leakGuard(holdoutTexts) {
+  const held = holdoutTexts.map(t => { const n = normalize(t); return { n, g: trigrams(n) }; });
+  return text => {
+    const n = normalize(text); const g = trigrams(n);
+    return held.some(h => {
+      const [short, long] = n.length <= h.n.length ? [n, h.n] : [h.n, n];
+      if (short.length > 12 && long.includes(short)) return true;
+      let inter = 0; for (const x of g) if (h.g.has(x)) inter++;
+      return inter / (g.size + h.g.size - inter) >= 0.5;
+    });
+  };
+}
+const splitFor = family => (parseInt(sha(family).slice(0, 8), 16) % 100 < VALIDATION_PERCENT ? 'validation' : 'train');
+
+function build({ home, appdata, holdoutTexts }) {
+  const raw = readSources({ home, appdata });
+  const dropped = { filtered: 0, duplicate: 0, leak: 0 };
+  const kept = raw.filter(r => keep(r.text) || (dropped.filtered++, false)).map(r => ({ ...r, text: redact(r.text.trim()) }));
+  const unique = dedupe(kept); dropped.duplicate = kept.length - unique.length;
+  const leaks = leakGuard(holdoutTexts);
+  const rows = unique.filter(r => !leaks(r.text) || (dropped.leak++, false))
+    .map(r => ({ id: sha(squash(r.text)).slice(0, 16), family: r.family, source: r.source, split: splitFor(r.family), text: r.text }));
+  const count = key => rows.reduce((a, r) => ((a[r[key]] = (a[r[key]] || 0) + 1), a), {});
+  return { rows, stats: { extracted: raw.length, kept: rows.length, dropped, bySource: count('source'), bySplit: count('split'),
+    families: new Set(rows.map(r => r.family)).size, zhShare: rows.length ? +(rows.filter(r => /[一-鿿]/.test(r.text)).length / rows.length).toFixed(3) : 0 } };
+}
+
+if (require.main === module) {
+  const args = process.argv.slice(2);
+  const opt = name => { const i = args.indexOf(name); return i >= 0 ? args[i + 1] : undefined; };
+  try {
+    const out = path.resolve(opt('--out') || path.join(os.homedir(), '.agents', 'harness-everything', 'system-one', 'training'));
+    const rel = path.relative(REPO, out);
+    if (!rel.startsWith('..') && !path.isAbsolute(rel)) throw new Error('refusing to write the dataset inside the repository');
+    const holdoutFile = opt('--holdout') || path.join(REPO, 'benchmarks', 'fixtures', 'system-one-holdout.json');
+    const holdoutTexts = JSON.parse(fs.readFileSync(holdoutFile, 'utf8')).cases.map(c => c.request.context);
+    const { rows, stats } = build({ home: opt('--home') || os.homedir(), appdata: opt('--appdata') || process.env.APPDATA || '', holdoutTexts });
+    fs.mkdirSync(out, { recursive: true });
+    const body = rows.map(r => JSON.stringify(r)).join('\n') + '\n';
+    fs.writeFileSync(path.join(out, 'prompts.jsonl'), body, { mode: 0o600 });
+    const report = { ...stats, sha256: sha(body) };
+    fs.writeFileSync(path.join(out, 'collect-stats.json'), `${JSON.stringify(report, null, 2)}\n`);
+    console.log(JSON.stringify(report));
+  } catch (err) {
+    console.error(`system-one-collect: ${err.message}`);
+    process.exitCode = 1;
+  }
+}
+module.exports = { readSources, keep, redact, dedupe, leakGuard, jaccard, splitFor, build };