harness-everything 0.28.0 → 0.30.0

package/benchmarks/results/system-one/laya-ft-2026-09-25/README.md

@@ -0,0 +1,83 @@
+# System One: fine-tuned Laya intent model, holdout promotion run (2026-09-25)
+
+First Phase 3 promotion run for [#255](https://github.com/dyphn1/Harness-everything/issues/233):
+multilingual Laya 322M fine-tuned with RLCD on 42,156 soft-target items
+(3,513 scored train prompts), evaluated on the frozen 222-case intent
+holdout through a research-only bridge (production transports untouched).
+It is not live-host evidence.
+**Result: the promotion gate fails. Graded accepted precision lands exactly
+on the line (0.850, strict gate reads 0.8499…), coverage is 6.3% against an
+80% target, and the tradeoff does not beat the intent n-gram baseline
+(95.6% at 7.2%). Per the #255 stop/go rule, no PRAXIST scale-up is started
+on this evidence.**
+
+## Model
+
+- **Base.** `convaiinnovations/laya` multilingual, revision `55cf4c4e`.
+- **Training.** Single-device RLCD port of the upstream recipe (GRPO +
+  soft cross-entropy, 2 epochs, local MPS fp32, ~3.1 h). Held-out
+  calibration slice, per-type temperatures, fp16 artifact.
+- **Bridge.** `harness-everything/scripts/system-one/laya_adapter.py`:
+  fixed v1 noul questions, 13th option (unclassified) on a linear abstain
+  ramp under cutoff 0.45, vector normalized to the contract simplex.
+  Resident serve machinery reused from `cua_adapter`; `resident.js` and
+  `provider.js` unchanged.
+- **Thresholds.** `minConfidence` 0.5, `minMargin` 0, secondary 0.05 from
+  `system-one-calibrate.js` on 727 validation scores (graded 0.927, exact
+  0.914 at 9.6% coverage).
+
+## Holdout (222 owner-reviewed cases, two runs each)
+
+[`report.json`](report.json) is the unmodified evaluator output.
+
+| Gate | Value | Target | Pass |
+| --- | --- | --- | --- |
+| Accepted precision, graded | 0.850 (float 0.8499…) | ≥ 0.85 | no |
+| Accepted precision, exact | 0.786 | — | — |
+| Coverage | 6.3% (14/222) | ≥ 0.8 | no |
+| Macro-F1 vs majority baseline | 0.080 vs 0.020 | ≥ baseline | yes |
+| Repeatability (2 runs) | 1.0 | == 1 | yes |
+| Warm p95 | 179 ms (MPS) | ≤ 100 ms | no |
+| Source provenance | research transport | pinned | no |
+
+## Reading
+
+- **Same profile as the n-gram, slightly worse.** The n-gram intent model
+  scores 95.6% graded at 7.2% coverage; fine-tuned Laya scores 85.0% at
+  6.3%. The fine-tune clearly learned (zero-shot was 14% exact), but the
+  precision/coverage tradeoff does not clear the baseline.
+- **Paradigm mismatch, not just weights.** Twelve independent relevance
+  scores are forced into a single-winner 13-simplex; normalization crushes
+  confident rows below the acceptance bar. A relevance-native router (per-
+  intent thresholds instead of top-1 confidence) is a #233 contract
+  question, not a #255 training question.
+- **Transport notes for Phase 7.** MPS inference is deterministic across
+  runs (repeatability 1.0) but too slow for the warm gate (179 ms vs
+  100 ms); the MLX port measured 7–13 ms on the same class of hardware.
+- **Stop/go.** STOP the scale-up path on this evidence. Legitimate next
+  lanes (each frozen on validation first, never tuned on this holdout):
+  question rewording, peakier contract mapping, relevance-native routing
+  in #233.
+
+## Shadow run: relevance-native gates (frozen taus, raw scores)
+
+First evaluation under `docs/system-one-intent-gates.md` with frozen
+validation-fit thresholds (never tuned on this holdout). Raw independent
+scores, no simplex. `shadow-report.json` holds metrics only.
+
+| Gate | Value | Pass |
+| --- | --- | --- |
+| micro-F1 | 0.507 | no (≥ 0.55) |
+| macro-F1 | 0.501 | yes |
+| per-intent floors | breach | no |
+| weak precision | 0.181 | no |
+| cardinality | 2.51 predicted | yes |
+| abstain rate | 0.014 | yes |
+| family consistency | model 0.47 vs gold 0.80 | no |
+| repeatability | 1.0, byte-identical runs | yes |
+| warm p95 | 182 ms (≤ 250 ms bar) | yes |
+
+Verdict: not promotable under the reviewed contract. Owner gold is
+stricter than the teacher proxy (validation CV micro-F1 was 0.60), and
+weak intents collapse further (refactor-class precision 0.35 → 0.18).
+The gates caught exactly what they were written to catch.

package/benchmarks/results/system-one/laya-ft-2026-09-25/report.json

@@ -0,0 +1,71 @@
+{
+  "schemaVersion": 1,
+  "issue": 255,
+  "phase": "3 fine-tuned promotion run (research transport)",
+  "model": "laya/harness-intent-s1-full (multilingual 322M RLCD, 42156 items, 2 epochs, local MPS)",
+  "calibration": {
+    "feasible": true,
+    "rows": 727,
+    "minPrecision": 0.85,
+    "task": "intent",
+    "secondaryThreshold": 0.05,
+    "minConfidence": 0.5,
+    "minMargin": 0,
+    "coverage": 0.09628610729023383,
+    "precision": 0.9271428571428573,
+    "accepted": 70,
+    "exactPrecision": 0.9142857142857143
+  },
+  "report": {
+    "schemaVersion": 1,
+    "cases": 222,
+    "baseline": {
+      "accuracy": 0.14864864864864866,
+      "macroF1": 0.01990950226244344,
+      "coverage": 1,
+      "abstentionRate": 0,
+      "acceptedPrecision": 0.14864864864864866,
+      "acceptedErrorRate": 0.8513513513513513
+    },
+    "model": {
+      "accuracy": 0.04954954954954955,
+      "macroF1": 0.07959346096997919,
+      "coverage": 0.06306306306306306,
+      "abstentionRate": 0.9369369369369369,
+      "acceptedPrecision": 0.8499999999999999,
+      "acceptedErrorRate": 0.21428571428571427,
+      "exactPrecision": 0.7857142857142857
+    },
+    "familyConsistency": {
+      "gold": {
+        "families": 83,
+        "consistent": 66,
+        "rate": 0.7951807228915663
+      },
+      "model": {
+        "families": 83,
+        "consistent": 75,
+        "rate": 0.9036144578313253
+      }
+    },
+    "agreement": 1,
+    "latency": {
+      "coldP95Ms": null,
+      "warmP95Ms": 178.71229199999652,
+      "coldSamples": 0,
+      "warmSamples": 444
+    },
+    "gates": {
+      "reviewedHoldout": true,
+      "acceptedPrecision": false,
+      "coverage": false,
+      "macroF1": true,
+      "repeatability": true,
+      "warmLatency": false,
+      "sourceProvenance": false,
+      "policyEvidence": false,
+      "liveHostEvidence": false
+    },
+    "rolloutReady": false
+  }
+}

package/benchmarks/results/system-one/laya-ft-2026-09-25/shadow-report.json

@@ -0,0 +1,44 @@
+{
+  "model": "laya/harness-intent-s1-full",
+  "thresholds": "frozen validation-fit taus (perintent-taus.json)",
+  "micro": {
+    "precision": 0.6068222621184919,
+    "recall": 0.43556701030927836,
+    "f1": 0.5071267816954239
+  },
+  "macroF1": 0.501,
+  "cardinality": {
+    "predicted": 2.509009009009009,
+    "gold": 3.4954954954954953
+  },
+  "abstainRate": 0.014,
+  "exactSetMatch": 0.068,
+  "weakPrecision": 0.181,
+  "family": {
+    "gold": {
+      "families": 83,
+      "consistent": 66,
+      "rate": 0.7951807228915663
+    },
+    "model": {
+      "families": 83,
+      "consistent": 39,
+      "rate": 0.46987951807228917
+    }
+  },
+  "gates": {
+    "microF1": false,
+    "macroF1": true,
+    "perIntentFloors": false,
+    "weakPrecision": false,
+    "cardinality": true,
+    "abstainRate": true,
+    "familyConsistency": false,
+    "repeatability": true,
+    "warmLatency": true,
+    "structuralControls": true,
+    "policyEvidence": false,
+    "liveHostEvidence": false,
+    "rolloutReady": false
+  }
+}

package/benchmarks/results/system-one/laya-zs-2026-09-25/README.md

@@ -0,0 +1,68 @@
+# System One: Laya zero-shot dense-label baseline (2026-09-25)
+
+Local zero-shot evidence for the intent stage
+([system-one-intent.md](../../../../docs/system-one-intent.md)) with Laya as a
+typed-decision substrate. It is not live-host evidence. Issue
+[#255](https://github.com/dyphn1/Harness-everything/issues/233) (research
+track for [#233](https://github.com/dyphn1/Harness-everything/issues/233)).
+**Result: zero-shot Laya is not a teacher. Primary exact agreement with the
+sonnet-scored labels is ~15% (graded ~0.26); per-intent calibration cuts score
+error by ~40% but adds almost no discrimination. Fine-tuning (Phase 3) or
+question rewording is required before any training use.**
+
+## Method
+
+- **Mapping.** Each of the 12 owner intents (catalog order, frozen) becomes one
+  Laya `noul` question (`scripts/system-one-label-laya.py`, questions v1).
+  Derived gold/secondary follow the owner bands mechanically and are marked
+  `derived: true`; they are bootstrap, never owner truth.
+- **Backends.** Official PyTorch Laya 0.3.20 on MPS and the independent MLX
+  port 0.2.0 on GPU, both over the 727 validation prompts (local only).
+- **Compared.** The 537 validation rows that also have sonnet-scored labels.
+  [`report.json`](report.json) holds aggregates, hashes and distributions only;
+  no prompt text leaves the private corpus.
+
+## Zero-shot vs teacher (537 rows)
+
+| Metric | torch | mlx |
+| --- | --- | --- |
+| Primary exact | 14.3% | 14.9% |
+| Graded mean | 0.261 | 0.267 |
+| Mean top score (over-confident) | 0.759 | 0.741 |
+| Null rate (teacher: 14.3%) | 3.5% | 3.9% |
+| zh slice exact / graded (n=495) | 13.7% / 0.259 | 14.5% / 0.266 |
+| en slice exact / graded (n=42) | 21.4% / 0.286 | 19.0% / 0.276 |
+
+Per-intent rank correlation with the teacher sits at 0.1–0.3 (best: fix/git
+~0.3; worst: explain/review/docs ~0.1).
+
+## Backend parity (same 537 rows)
+
+Overall agreement is poor (routing 61.6%, derived gold 63.1%) **entirely
+because of routing**: the MLX port sends ~250 more Chinese rows to the English
+checkpoint than the official Router (torch 441 multilingual / 96 english vs
+mlx 237 / 300). On the 331 rows routed to the same checkpoint, numeric parity
+is excellent: mean max |Δp| 0.0055, p95 0.0134, derived-gold agreement 98.2%.
+Lesson: pin the checkpoint per language for bulk labeling; do not rely on
+router agreement across implementations.
+
+## Calibration (5-fold CV, per-intent isotonic)
+
+| Metric | torch | mlx |
+| --- | --- | --- |
+| CV exact | 19.4% | 22.5% |
+| CV graded | 0.236 | 0.267 |
+| CV MAE (raw → calibrated) | 0.323 → 0.193 | 0.304 → 0.194 |
+
+Calibration fixes the scale but cannot create rank signal. The zero-shot +
+calibration path stays far below any promotion gate.
+
+## Reading
+
+- **Not a teacher.** Zero-shot scores correlate weakly with owner-judged
+  relevance; the bands derive mostly noise.
+- **MLX is a viable fast path** once the checkpoint is pinned per language
+  (98% agreement with the official backend on shared checkpoints).
+- **Next levers (Phase 3).** Soft-target fine-tuning on the 3,513 scored
+  train rows; question rewording as a cheap parallel lane. PRAXIST enters at
+  Phase 5, after a standalone fine-tune baseline exists.

package/benchmarks/results/system-one/laya-zs-2026-09-25/report.json

@@ -0,0 +1,173 @@
+{
+  "schemaVersion": 1,
+  "issue": 255,
+  "phase": "1+2 zero-shot baseline",
+  "compared": 537,
+  "teacher": "labels-intent-scores.jsonl",
+  "teacherSha256": "c25c4d9e7ae3f021b66c64b5b0b7dfe7d044539860a7a7efe6be5a91f77b7618",
+  "promptsSha256": "54d1446ddb1e71948e33be86c37147f4668addedaf782bfe6c2754bd7d9e9e4f",
+  "questions": "v1 (12 noul, catalog order)",
+  "provenance": {
+    "torch": {
+      "laya": "0.3.20",
+      "torch": "2.14.0",
+      "transformers": "5.17.0",
+      "python": "3.12.13",
+      "device": "mps",
+      "latencyMs": {
+        "p50": 134.74925002083182,
+        "p95": 428.49976690486073
+      }
+    },
+    "mlx": {
+      "laya-mlx": "0.2.0",
+      "mlx": "0.32.2",
+      "python": "3.12.13",
+      "device": "gpu",
+      "note": "independent third-party port, inference only"
+    },
+    "host": "Mac mini M4, 16GB, macOS 26.6.2"
+  },
+  "candidates": {
+    "labels-intent-laya-zs-torch.jsonl": {
+      "sha256": "79c9e2a87b1186f2707ad2a1861492ffaf8400c1fa3a374ad8d5bc48ecccc83b",
+      "n": 537,
+      "primaryExact": 0.14338919925512103,
+      "gradedMean": 0.26126629422718806,
+      "meanTopScore": 0.7589113594040968,
+      "meanScoreByIntent": {
+        "explain": 0.49797858472998136,
+        "discuss": 0.1724169459962756,
+        "git": 0.22799664804469275,
+        "fix": 0.30267448789571694,
+        "edit": 0.4249581005586592,
+        "feature": 0.4799590316573557,
+        "refactor": 0.503630540037244,
+        "review": 0.3949202979515829,
+        "test": 0.4052284916201117,
+        "docs": 0.2692115456238361,
+        "plan": 0.24817392923649906,
+        "investigate": 0.22743109869646183
+      },
+      "rankCorrByIntent": {
+        "explain": 0.027784640172690576,
+        "discuss": 0.23116435330770707,
+        "git": 0.29041398684548636,
+        "fix": 0.29932632635456186,
+        "edit": 0.2279588743991616,
+        "feature": 0.2573118860847266,
+        "refactor": 0.11193843868650764,
+        "review": 0.08718817389449918,
+        "test": 0.16812905421595062,
+        "docs": 0.0939364567373099,
+        "plan": 0.16190725568849487,
+        "investigate": 0.1642989079441322
+      },
+      "nullRate": 0.035381750465549346,
+      "teacherNullRate": 0.14338919925512103,
+      "slices": {
+        "zh": {
+          "n": 495,
+          "exact": 0.13737373737373737,
+          "graded": 0.25919191919191914
+        },
+        "en": {
+          "n": 42,
+          "exact": 0.21428571428571427,
+          "graded": 0.2857142857142857
+        }
+      },
+      "routing": {
+        "multilingual": 441,
+        "english": 96
+      }
+    },
+    "labels-intent-laya-zs-mlx.jsonl": {
+      "sha256": "bd8488b0436373b16c00067e66b2e583299700e5b622e5cdd8c8983f3b8ab5e4",
+      "n": 537,
+      "primaryExact": 0.148975791433892,
+      "gradedMean": 0.26685288640595906,
+      "meanTopScore": 0.7410662942271881,
+      "meanScoreByIntent": {
+        "explain": 0.31080893854748604,
+        "discuss": 0.16107765363128493,
+        "git": 0.2408,
+        "fix": 0.34066741154562385,
+        "edit": 0.37259962756052145,
+        "feature": 0.453928677839851,
+        "refactor": 0.5032795158286778,
+        "review": 0.416868156424581,
+        "test": 0.38708659217877095,
+        "docs": 0.24436666666666665,
+        "plan": 0.25952569832402234,
+        "investigate": 0.23216275605214154
+      },
+      "rankCorrByIntent": {
+        "explain": 0.15066239073164425,
+        "discuss": 0.17409543733224803,
+        "git": 0.2796312099034226,
+        "fix": 0.2399448780065943,
+        "edit": 0.2460774826207481,
+        "feature": 0.2812117477450446,
+        "refactor": 0.13321742515847998,
+        "review": 0.06277577657598903,
+        "test": 0.13653433833924808,
+        "docs": 0.2028910451790806,
+        "plan": 0.1757922914760996,
+        "investigate": 0.12610523734880044
+      },
+      "nullRate": 0.03910614525139665,
+      "teacherNullRate": 0.14338919925512103,
+      "slices": {
+        "zh": {
+          "n": 495,
+          "exact": 0.14545454545454545,
+          "graded": 0.2660606060606061
+        },
+        "en": {
+          "n": 42,
+          "exact": 0.19047619047619047,
+          "graded": 0.2761904761904762
+        }
+      },
+      "routing": {
+        "english": 300,
+        "multilingual": 237
+      }
+    }
+  },
+  "parity": {
+    "pair": [
+      "labels-intent-laya-zs-torch.jsonl",
+      "labels-intent-laya-zs-mlx.jsonl"
+    ],
+    "n": 537,
+    "routingAgreement": 0.6163873370577281,
+    "derivedGoldAgreement": 0.6312849162011173,
+    "maxAbsDeltaMean": 0.2839195530726257,
+    "maxAbsDeltaP95": 0.9104
+  },
+  "sameRouteParity": {
+    "n": 331,
+    "maxAbsDeltaMean": 0.0055,
+    "maxAbsDeltaP95": 0.0134,
+    "maxAbsDeltaMax": 0.1711,
+    "derivedGoldAgreement": 0.982
+  },
+  "calibrationCV": {
+    "torch": {
+      "exact": 0.194,
+      "graded": 0.236,
+      "mae": 0.1929,
+      "rawMAE": 0.3226,
+      "bestCutoff": "0.20"
+    },
+    "mlx": {
+      "exact": 0.225,
+      "graded": 0.267,
+      "mae": 0.1941,
+      "rawMAE": 0.3036,
+      "bestCutoff": "0.00"
+    }
+  }
+}