harness-everything 0.29.0 → 0.30.1

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (108) hide show
  1. package/.claude-plugin/marketplace.json +1 -1
  2. package/.claude-plugin/plugin.json +1 -1
  3. package/AGENTS.md +1 -0
  4. package/CHANGELOG.md +42 -0
  5. package/PRIVACY.md +9 -0
  6. package/benchmarks/fixtures/system-one-intent-holdout-draft.json +2228 -0
  7. package/benchmarks/fixtures/system-one-intent-holdout.json +16509 -0
  8. package/benchmarks/fixtures/system-one-intent-reviews.json +2723 -0
  9. package/benchmarks/results/system-one/intent-ngram-v1-2026-09-24/README.md +64 -0
  10. package/benchmarks/results/system-one/intent-ngram-v1-2026-09-24/report.json +15177 -0
  11. package/benchmarks/results/system-one/labeler-holdout-check-intent-2026-09-24/report.json +2976 -0
  12. package/benchmarks/results/system-one/labeler-holdout-check-intent-scores-haiku-2026-09-24/report-005-steps.json +6125 -0
  13. package/benchmarks/results/system-one/labeler-holdout-check-intent-scores-haiku-2026-09-24/report-haiku-sonnet-escalation.json +6344 -0
  14. package/benchmarks/results/system-one/labeler-holdout-check-intent-scores-haiku-2026-09-24/report-scaled-margin-escalation.json +6206 -0
  15. package/benchmarks/results/system-one/labeler-holdout-check-intent-scores-haiku-2026-09-24/report.json +6103 -0
  16. package/benchmarks/results/system-one/laya-ft-2026-09-25/README.md +83 -0
  17. package/benchmarks/results/system-one/laya-ft-2026-09-25/report.json +71 -0
  18. package/benchmarks/results/system-one/laya-ft-2026-09-25/shadow-report.json +44 -0
  19. package/benchmarks/results/system-one/laya-zs-2026-09-25/README.md +68 -0
  20. package/benchmarks/results/system-one/laya-zs-2026-09-25/report.json +173 -0
  21. package/ci/fable-mode-test.js +16 -0
  22. package/ci/mechanism-34-observations.test.js +550 -0
  23. package/ci/mechanism-system-one-collect.test.js +16 -0
  24. package/ci/mechanism-system-one-intent.test.js +406 -0
  25. package/ci/mechanism-system-one-laya.test.js +45 -0
  26. package/ci/mechanism-system-one-relevance.test.js +112 -0
  27. package/ci/mechanism-system-one-resident.test.js +16 -0
  28. package/ci/system-one-laya-calibrate-test.py +65 -0
  29. package/ci/system-one-laya-export-test.py +155 -0
  30. package/ci/system-one-laya-label-test.py +201 -0
  31. package/ci/system-one-laya-perintent-test.py +60 -0
  32. package/ci/system-one-laya-provider-test.py +115 -0
  33. package/ci/system-one-laya-train-test.py +93 -0
  34. package/docs/routing.md +25 -0
  35. package/docs/system-one-intent-gates.md +102 -0
  36. package/docs/system-one-intent.md +173 -0
  37. package/docs/system-one-laya-training.md +38 -0
  38. package/docs/system-one-observations.md +206 -0
  39. package/docs/system-one-routing.md +25 -10
  40. package/docs/system-one-suggestion-gates.md +292 -0
  41. package/docs/system-one-training.md +56 -6
  42. package/docs/workflows/fable-mode.md +7 -0
  43. package/fable-mode/SKILL.md +1 -1
  44. package/fable-mode/agents/fable-orchestrator.md +4 -4
  45. package/fable-mode/agents/fable-worker-haiku.md +4 -4
  46. package/fable-mode/agents/fable-worker-sonnet.md +4 -4
  47. package/fable-mode/execution-guardrails/SKILL.md +5 -3
  48. package/fable-mode/execution-guardrails/references/guardrail-rules.md +12 -8
  49. package/fable-mode/fable-haiku/SKILL.md +1 -1
  50. package/fable-mode/fable-opus/SKILL.md +1 -1
  51. package/fable-mode/fable-sonnet/SKILL.md +1 -1
  52. package/harness-everything/SKILL.md +1 -1
  53. package/harness-everything/scripts/kernel-router.js +10 -0
  54. package/harness-everything/scripts/system-one/catalogs.js +35 -0
  55. package/harness-everything/scripts/system-one/contract.js +45 -1
  56. package/harness-everything/scripts/system-one/evaluate.js +115 -19
  57. package/harness-everything/scripts/system-one/intent.js +22 -0
  58. package/harness-everything/scripts/system-one/laya_adapter.py +141 -0
  59. package/harness-everything/scripts/system-one/provider.js +10 -3
  60. package/harness-everything/scripts/system-one/resident.js +8 -3
  61. package/harness-everything/scripts/system-one/router.js +1 -1
  62. package/hooks/hooks.json +40 -0
  63. package/hooks/scripts/codex-user-prompt.js +39 -0
  64. package/hooks/scripts/lib/observations.js +581 -0
  65. package/hooks/scripts/observation-hook.js +25 -0
  66. package/opencode-plugin/plugin.json +1 -1
  67. package/package.json +1 -1
  68. package/plugins/harness-everything/.codex-plugin/plugin.json +1 -1
  69. package/plugins/harness-everything/hooks/hooks.json +35 -2
  70. package/plugins/harness-everything/hooks/scripts/codex-user-prompt.js +39 -0
  71. package/plugins/harness-everything/hooks/scripts/lib/observations.js +581 -0
  72. package/plugins/harness-everything/hooks/scripts/observation-hook.js +25 -0
  73. package/plugins/harness-everything/plugin.json +1 -1
  74. package/plugins/harness-everything/skills/fable-mode/SKILL.md +1 -1
  75. package/plugins/harness-everything/skills/fable-mode/agents/fable-orchestrator.md +4 -4
  76. package/plugins/harness-everything/skills/fable-mode/agents/fable-worker-haiku.md +4 -4
  77. package/plugins/harness-everything/skills/fable-mode/agents/fable-worker-sonnet.md +4 -4
  78. package/plugins/harness-everything/skills/fable-mode/execution-guardrails/SKILL.md +5 -3
  79. package/plugins/harness-everything/skills/fable-mode/execution-guardrails/references/guardrail-rules.md +12 -8
  80. package/plugins/harness-everything/skills/fable-mode/fable-haiku/SKILL.md +1 -1
  81. package/plugins/harness-everything/skills/fable-mode/fable-opus/SKILL.md +1 -1
  82. package/plugins/harness-everything/skills/fable-mode/fable-sonnet/SKILL.md +1 -1
  83. package/plugins/harness-everything/skills/harness-everything/SKILL.md +1 -1
  84. package/plugins/harness-everything/skills/harness-everything/scripts/kernel-router.js +10 -0
  85. package/plugins/harness-everything/skills/harness-everything/scripts/system-one/catalogs.js +35 -0
  86. package/plugins/harness-everything/skills/harness-everything/scripts/system-one/contract.js +45 -1
  87. package/plugins/harness-everything/skills/harness-everything/scripts/system-one/evaluate.js +115 -19
  88. package/plugins/harness-everything/skills/harness-everything/scripts/system-one/intent.js +22 -0
  89. package/plugins/harness-everything/skills/harness-everything/scripts/system-one/laya_adapter.py +141 -0
  90. package/plugins/harness-everything/skills/harness-everything/scripts/system-one/provider.js +10 -3
  91. package/plugins/harness-everything/skills/harness-everything/scripts/system-one/resident.js +8 -3
  92. package/plugins/harness-everything/skills/harness-everything/scripts/system-one/router.js +1 -1
  93. package/sbom.cdx.json +7 -7
  94. package/scripts/evaluate-system-one.js +17 -5
  95. package/scripts/system-one-calibrate-laya.py +165 -0
  96. package/scripts/system-one-calibrate.js +42 -13
  97. package/scripts/system-one-collect.js +7 -3
  98. package/scripts/system-one-compare-laya.py +156 -0
  99. package/scripts/system-one-corpus.js +22 -15
  100. package/scripts/system-one-eval-laya.js +112 -0
  101. package/scripts/system-one-export-laya.py +197 -0
  102. package/scripts/system-one-label-laya.py +243 -0
  103. package/scripts/system-one-label.js +163 -37
  104. package/scripts/system-one-observations-export.js +106 -0
  105. package/scripts/system-one-observations-review.js +86 -0
  106. package/scripts/system-one-perintent-laya.py +134 -0
  107. package/scripts/system-one-train-laya.py +344 -0
  108. package/scripts/system-one-train-ngram.py +66 -14
@@ -42,10 +42,16 @@ The manifest is version 1 with `python` (executable), `checkpoint` (absolute
42
42
  `.safetensors` path), `weightsSha256`, `configSha256` (matching `.json` sidecar),
43
43
  `modelId`, `revision`, `domain`, and optional `timeoutMs` (1–10000, default 2000).
44
44
  All fields are required except the timeout; unknown fields are rejected. An
45
- optional `acceptance` object holds exactly `minConfidence` (0.5–0.99) and
46
- `minMargin` (0–0.9). These are the thresholds calibrated on validation for this
45
+ optional `acceptance` object holds `minConfidence` (0.5–0.99) and
46
+ `minMargin` (0–0.9). It may also hold `secondaryThreshold` (0.05–0.5), the
47
+ probability at which another intent counts as a secondary intent; that key is
48
+ valid only for an intent checkpoint ([system-one-intent.md](system-one-intent.md#scoring)).
49
+ These are the thresholds calibrated on validation for this
47
50
  checkpoint. A scored result carries them, and the router and the evaluator pass
48
- them to `decide`. Without the field the Phase 0 defaults apply. Hashes
51
+ the first two to `decide`. Without the field the Phase 0 defaults apply. For an
52
+ intent corpus the evaluator also reports graded accepted precision, exact accepted
53
+ precision and family consistency. The `acceptedPrecision` gate uses the graded
54
+ value. Hashes
49
55
  must be lowercase SHA-256. Revision and domain are artifact-owner declarations,
50
56
  not independently verified quality claims. The known `cua-ai/cua-s1-forms` ID
51
57
  must use domain `forms-v1`. No pickle format or remote URL is accepted.
@@ -349,23 +355,32 @@ small model only has to separate a few options at a time.
349
355
 
350
356
  1. **Tier**: `tier1`, `tier2`, `tier3` or `unclassified`, following
351
357
  [system-one-corpus.md](system-one-corpus.md).
352
- 2. **Intent**: the kind of work the prompt asks for, drawn from a small fixed
353
- catalog (for example: ask or explain, discuss or decide, Git/GitHub
354
- operation, fix a bug, build a feature, refactor, review or audit, test or
355
- verify, write docs, plan or spec, investigate). The intent catalog is the
356
- owner's and is recorded in the corpus document before any intent labels.
358
+ 2. **Intent**: the kind of work the prompt asks for, drawn from the owner's
359
+ fixed catalog in [system-one-intent.md](system-one-intent.md): `explain`,
360
+ `discuss`, `git`, `fix`, `edit`, `feature`, `refactor`, `review`, `test`,
361
+ `docs`, `plan`, `investigate` or `unclassified`. A prompt has one primary
362
+ intent and may have secondary intents.
357
363
  3. **Skills**: which canonical skills fit the prompt, scored against skill
358
364
  descriptions. Only skills above the stage's calibrated threshold are
359
365
  suggested, and they are suggestions, never required reads.
360
366
 
361
- Each stage has its own gates, and a stage ships only when it passes them. Tier
362
- comes first because the reviewed holdout exists for it. Later stages reuse the
367
+ Each stage has its own gates, and a stage ships only when it passes them.
368
+ The relevance-native plan in
369
+ [system-one-suggestion-gates.md](system-one-suggestion-gates.md#readout-per-stage)
370
+ adds a validity stage first and runs intent before tier, because tier3 is
371
+ composed from the intent scores. Tier was trained first because the
372
+ reviewed holdout exists for it. Later stages reuse the
363
373
  same collector, labeler, trainer and evaluator with a different catalog.
364
374
  Suggestions are advisory: explicit workflow requests, action gates, memory
365
375
  ownership, the Rule of 3 and deterministic policy always take precedence.
366
376
 
367
377
  ## Rollout gates
368
378
 
379
+ Model-quality gates for relevance-native candidates are defined in
380
+ [system-one-suggestion-gates.md](system-one-suggestion-gates.md), which
381
+ supersedes the precision/coverage/latency targets below. The holdout size,
382
+ provenance, repeatability and advisory rules below still apply.
383
+
369
384
  Phase 4 requires a separately reviewed, family-disjoint holdout of at least 200
370
385
  cases, at least 50 each in English and Traditional Chinese. System One output is
371
386
  advisory: it informs the agent and suggests skills. It never grants or removes
@@ -0,0 +1,292 @@
1
+ # System One suggestion gates
2
+
3
+ Status: **proposed, pending owner review**. Once the owner approves it, this
4
+ document is frozen for the next candidate of each stage. Decided in the
5
+ [#233 design correction](https://github.com/dyphn1/Harness-everything/issues/233#issuecomment-5842359628)
6
+ (one event, matched against every option); research evidence is in
7
+ [#255](https://github.com/dyphn1/Harness-everything/issues/255).
8
+
9
+ This document supersedes the table in
10
+ [system-one-intent-gates.md](system-one-intent-gates.md). That table stays
11
+ as the historical contract for the runs made under it. It also supersedes
12
+ the model-quality part of
13
+ [system-one-routing.md#rollout-gates](system-one-routing.md#rollout-gates).
14
+ `evaluate.relevanceGates` still implements the superseded table. The code
15
+ change follows this document (docs → RED → GREEN) in its own commit.
16
+
17
+ ## Principle
18
+
19
+ 1. **One event, scored against every option.** For a prompt, the scorer
20
+ returns an independent relevance `p_i ∈ [0, 1]` for every option of the
21
+ stage catalog. Scores do not sum to 1 and are never renormalized into a
22
+ simplex. With CUA-S1 this is the per-option logit passed through a
23
+ sigmoid. Upstream `tinyx` already encodes each option separately and
24
+ scores it against the context, so one forward pass scores every option.
25
+ 2. **Suggestions, not verdicts.** Output is advice to the agent. A wrong
26
+ suggestion costs little; a confident wrong suggestion, or a long list the
27
+ agent must read, costs trust and tokens. Gates therefore measure
28
+ *trustworthy confidence* and *suggestion coverage under a noise cap*,
29
+ not exact accuracy.
30
+ 3. **Deterministic controls stay outside learned scoring.** Tier structural
31
+ floor, explicit workflow requests, action/security approval, scope and
32
+ memory ownership and the Rule of 3 are unchanged. Abstention always falls
33
+ back to the lexical path.
34
+
35
+ ## Readout per stage
36
+
37
+ All stages share the scoring and the calibration. Only the readout differs.
38
+
39
+ | Stage | Catalog | Readout | Cap |
40
+ | --- | --- | --- | --- |
41
+ | Validity | `actionable`, `invalid` | `invalid` at or above its threshold → stop: no tier, workflow, intent or skill suggestion; the host agent decides | — |
42
+ | Intent | the 12 intents of [system-one-intent.md](system-one-intent.md) | set: every intent at or above its threshold, highest `p` first, truncated at the cap | 3 |
43
+ | Tier | `tier1`, `tier2`, `tier3` | composed (see [Tier composition](#tier-composition)): tier3 when `feature` or `refactor` reaches its intent threshold; otherwise the tier scorer's pick; final tier = max(that, structural floor); none → abstain | 1 |
44
+ | Workflow | the router's strategies (`direct-single`, `iterative-single`, `fable-staged`, `fable-parallel`, `fable-multi-agent-workspace`) | pick one, same rule; an explicit workflow request always wins | 1 |
45
+ | Skills | canonical skills, scored against their descriptions | set, same rule | 3 |
46
+
47
+ The cap of 3 for set stages comes from AGENTS.md rule 10: a suggested skill
48
+ must be read before it can be dismissed, so each extra suggestion has a
49
+ reading cost. The cap is part of the readout, so noise is bounded by
50
+ construction; the gates then judge what survives the cap.
51
+
52
+ Stages run in table order: validity, intent, tier, workflow, skills. Tier
53
+ reads the intent scores, so intent runs before it.
54
+
55
+ ### Tier composition
56
+
57
+ Tier rules v2 (owner decision, 2026-09-26): tier3 covers new features,
58
+ refactors, changes to definitions, and cross-repository or cross-component
59
+ work, whatever the width of the change; tier1 is a clear bounded action
60
+ (Git, status, lookup, direct answer); tier2 is a bounded modification.
61
+
62
+ Whether a prompt asks for a feature or a refactor is an intent question,
63
+ and the intent stage already scores it. The tier stage therefore does not
64
+ learn that part again:
65
+
66
+ 1. **Intent part.** tier3 when the calibrated score of `feature` or
67
+ `refactor` is at or above that intent's threshold (either one fires; not
68
+ "is the top intent", which a small intent scorer rarely satisfies).
69
+ 2. **Scope part.** The deterministic structural floor (cross-repository,
70
+ cross-component, macro scope) sets tier3 as before.
71
+ 3. **Tier scorer.** Otherwise the tier scorer picks, trained on the
72
+ breadth-based labels, so it separates tier1 from tier2 and keeps the
73
+ breadth-based tier3.
74
+
75
+ Over-reading `feature`/`refactor` moves a prompt one tier up, which the
76
+ gates count as acceptable; missing it moves it down, which they count as
77
+ under-tier. The composition is therefore tuned for intent recall on those
78
+ two intents.
79
+
80
+ Evidence (#265, validation, five seeds): composed with the CUA-S1 intent
81
+ scorer, acceptable precision 0.82 and under-tier 0.13, against 0.77 and
82
+ 0.22 for a tier scorer trained directly on rules-v2 labels.
83
+
84
+ **Holdout.** The owner decided not to re-review the tier holdout under
85
+ rules v2. Its gold follows the older rules, and rules v2 only moves labels
86
+ up (tier1 or tier2 to tier3). So underRate on the old gold stays valid, and
87
+ a tier3 pick for tier2 gold counts as acceptable. A tier3 pick for a
88
+ tier1-gold feature prompt counts as an error, which makes
89
+ acceptablePrecision slightly pessimistic; the count of such cases is
90
+ reported beside it.
91
+
92
+ `unclassified`/`null` is not an option in the new readout. It is the
93
+ abstain outcome: no option reaches its threshold.
94
+
95
+ **The scorer only sees the current prompt.** It never receives earlier
96
+ turns. A prompt whose text alone carries no actionable request is
97
+ `invalid`: System One makes no suggestion, and the host agent, which has
98
+ the conversation, decides. Validity runs first; the later stages run only
99
+ on `actionable` prompts.
100
+
101
+ ## Calibration and thresholds
102
+
103
+ - **Calibration.** A monotone per-option map (isotonic or Platt, chosen and
104
+ recorded before fitting) from raw `p` to calibrated `p`, fitted on
105
+ validation only. Needed because a model trained on soft teacher scores
106
+ outputs a *relevance degree* (0.5 = "secondary relevance"), not a
107
+ probability of being relevant. Gates read calibrated `p`.
108
+ - **Suggest threshold `τ_i`.** One per option, fitted on validation
109
+ (max-F1 against the validation truth), 2-fold CV reported.
110
+ - **Confident band.** Calibrated `p ≥ 0.8`. A confident suggestion is a
111
+ claim that at least 8 in 10 such suggestions are relevant.
112
+ - Calibration maps, thresholds, cap, band and this document are frozen
113
+ before a holdout run. Nothing is refit after it.
114
+
115
+ ## Truth
116
+
117
+ | Split | Relevant set | Primary |
118
+ | --- | --- | --- |
119
+ | Train / validation | teacher score ≥ 0.4, after owner intent overrides and exclusions | teacher primary |
120
+ | Intent holdout | owner `gold ∪ secondary` | owner `gold` |
121
+ | Tier holdout | owner `gold` tier; `null` means abstention is correct | — |
122
+
123
+ ### Invalid prompts
124
+
125
+ Some prompts cannot be judged from their text: a go-ahead (`go`, `好`,
126
+ `完成了`), an option pick, a change verb with no object (`修正一下`),
127
+ feedback on earlier work (`還是一樣的錯誤`), a pointer to earlier content
128
+ (`請依照需求實作`), a paste with no request, or chatter. The small model
129
+ does not try to recover the missing context. It marks them `invalid` and
130
+ hands them to the host agent (the external LLM), which sees the whole
131
+ conversation. Deterministic controls (action gates, scope and memory
132
+ ownership, the Rule of 3) apply unchanged. The noise-gate experiment (#265)
133
+ showed that tier labels given to such prompts from session context were
134
+ what kept a text-only model unsure.
135
+
136
+ - **Definition.** A holdout case is invalid when the frozen rules
137
+ (`rule_v2` in `scripts/system-one-noise-experiment.py`, #265) match its
138
+ text or its owner gold is `null`. The owner reviews that list before the
139
+ holdout run; the list is then frozen with the rest of step 4 below.
140
+ - **Scoring.** On an invalid case, the model is correct when it returns
141
+ `invalid` (or abstains); any tier, intent or skill suggestion is an error
142
+ for that case. Invalid cases are excluded from the later stages' metrics
143
+ and reported as invalid recall. On actionable cases, returning `invalid`
144
+ is reported as invalid false-positive rate.
145
+ - **Size.** The rules mark 9 of 215 tier-holdout cases (6 `null`, 2 `tier1`,
146
+ 1 `tier2`) and 3 of 222 intent-holdout cases; the tier holdout has 20
147
+ `null` cases in all. The holdouts were built with most continuations
148
+ removed, so the validity gate is read on validation until a reviewed
149
+ invalid set exists.
150
+
151
+ ## Metrics
152
+
153
+ Set stages (intent, skills), over actionable prompts:
154
+
155
+ | Metric | Definition |
156
+ | --- | --- |
157
+ | suggestionCoverage | share of prompts with at least one suggestion |
158
+ | confidentCoverage | share of prompts with at least one confident suggestion |
159
+ | primaryHit | share of prompts whose primary is among the suggestions |
160
+ | suggestionPrecision | relevant suggestions / all suggestions |
161
+ | confidentPrecision | relevant confident suggestions / all confident suggestions |
162
+ | ECE | expected calibration error over all (prompt, option) pairs, 10 equal-width bins of calibrated `p` against set membership |
163
+ | Brier | mean squared error of calibrated `p` against set membership |
164
+ | per-option precision / recall / support | support = holdout prompts where the option is relevant |
165
+ | cardinality | mean and p95 suggestions per prompt, share of prompts at the cap |
166
+ | recall | relevant suggestions / relevant options (reported) |
167
+
168
+ ECE alone rewards a timid model that never says anything confident; it is
169
+ gated together with confidentCoverage, never alone.
170
+
171
+ Pick stages (tier, workflow):
172
+
173
+ | Metric | Definition |
174
+ | --- | --- |
175
+ | coverage | share of actionable prompts that get a pick |
176
+ | acceptablePrecision | picks equal to gold or one tier above it, among picks |
177
+ | underRate | picks below gold, among picks |
178
+ | exactPrecision | picks equal to gold, among picks (reported) |
179
+ | ECE / Brier | as for set stages, one-vs-rest per option |
180
+
181
+ One tier above gold is acceptable because over-routing adds process;
182
+ under-routing skips obligations, so it has its own hard gate.
183
+
184
+ Every stage also reports family consistency (same family, same top
185
+ option), repeatability, latency, artifact size and policy evidence.
186
+
187
+ ## Gates
188
+
189
+ ### Intent (holdout)
190
+
191
+ Anchors are validation-proxy measurements, not holdout results: the
192
+ fine-tuned Laya reference, capped at 3 with in-sample thresholds, reaches
193
+ coverage 0.97, primary hit 0.75, precision 0.63, raw-`p ≥ 0.6` precision
194
+ 0.85 and ECE 0.06. Holdout truth is stricter, so bars sit below it.
195
+
196
+ | Gate | Bar |
197
+ | --- | --- |
198
+ | suggestionCoverage | ≥ 0.90 |
199
+ | primaryHit | ≥ 0.65 |
200
+ | suggestionPrecision | ≥ 0.60 |
201
+ | confidentCoverage | ≥ 0.40 |
202
+ | confidentPrecision | ≥ 0.75 (the 0.8 claim minus about two standard errors at about 250 confident suggestions) |
203
+ | ECE | ≤ 0.10 |
204
+ | per-intent precision, support ≥ 20 | ≥ 0.40 |
205
+ | family consistency (top suggestion) | ≥ 0.70 (owner gold 0.80) |
206
+
207
+ ### Tier (holdout)
208
+
209
+ | Gate | Bar |
210
+ | --- | --- |
211
+ | acceptablePrecision | ≥ 0.85 (the owner's advisory bar, now counting one tier above gold) |
212
+ | underRate | ≤ 0.05 |
213
+ | coverage | ≥ 0.60 |
214
+ | ECE | ≤ 0.10 |
215
+ | invalid recall (invalid cases) | reported; ≥ 0.80 on validation invalid rows |
216
+ | invalid false-positive rate (actionable cases) | ≤ 0.05 |
217
+
218
+ Tier bars have no validation anchor from a relevance-native tier model
219
+ yet. The first tier candidate reports its validation numbers first, and
220
+ these bars are confirmed or amended in this document **before** its
221
+ holdout run, never after.
222
+
223
+ ### Workflow and skills
224
+
225
+ No reviewed holdout exists for these stages, so they cannot be promoted.
226
+ Before any gate applies, each needs:
227
+
228
+ - a fixed catalog document (skills: option text written for scoring; the
229
+ current 96-byte option limit truncates skill descriptions, so the
230
+ checkpoint's `option_tokens` must be sized for them);
231
+ - an owner-reviewed, family-disjoint holdout of at least 200 cases, at
232
+ least 50 each in English and Traditional Chinese;
233
+ - the paired lexical baseline on that holdout.
234
+
235
+ Their gates take the shape of tier (workflow) and intent (skills). A
236
+ learned stage must also be no worse than the lexical baseline on
237
+ primaryHit/suggestionPrecision (skills) or acceptablePrecision (workflow)
238
+ in the same paired run.
239
+
240
+ ### Every stage
241
+
242
+ | Gate | Bar |
243
+ | --- | --- |
244
+ | repeatability | identical scores and suggestions across two runs |
245
+ | warm p95 | ≤ 250 ms, with the host record ([latency note](system-one-intent-gates.md#latency-note)) |
246
+ | deployable artifact | CPU only, checkpoint ≤ 10 MB, installed by `npm run system-one:install` with pinned hashes |
247
+ | structural controls | unchanged, rollback path live |
248
+ | policyEvidence / liveHostEvidence | reported; required before the default changes from `shadow` |
249
+
250
+ The artifact gate encodes the #255 decision: Laya is a teacher and a
251
+ ceiling reference, never a production candidate.
252
+
253
+ ## Evaluation order
254
+
255
+ 1. Train on train rows only.
256
+ 2. Fit calibration and thresholds on validation (2-fold CV reported).
257
+ 3. **Qualify on validation.** Report every metric above against the
258
+ validation truth. A candidate may spend a holdout run only if it meets
259
+ the holdout bars on validation *and* passes the deployable-artifact
260
+ gate. Otherwise it stays a validation result.
261
+ 4. Freeze the checkpoint hash, calibration, thresholds, evaluator
262
+ revision and this document.
263
+ 5. Score the holdout once, with two runs for repeatability.
264
+ 6. A fail is final for that candidate. A new candidate restarts at step 1
265
+ with a recorded reason.
266
+
267
+ ## Holdout budget
268
+
269
+ Every holdout run spends the holdout a little: each published result makes
270
+ later choices less independent. Each run is recorded in the evidence
271
+ directory with candidate hash, date and aggregate result.
272
+
273
+ - Intent holdout, spent before this document: n-gram intent v1,
274
+ fine-tuned Laya (single-winner), fine-tuned Laya (relevance shadow).
275
+ - Tier holdout, spent before this document: tinyx v1 and the n-gram
276
+ versions reported in #233, plus the lexical baseline.
277
+
278
+ Under this document, each holdout allows **one** further candidate run.
279
+ Before a second run, the owner adds at least 50 freshly reviewed cases (at
280
+ least 20 per language) that no earlier candidate or threshold saw, and the
281
+ gates are read on the enlarged holdout.
282
+
283
+ Earlier holdout results are historical. They are never re-scored under
284
+ these gates to claim a pass.
285
+
286
+ ## Non-goals
287
+
288
+ - The single-winner path (`contract.decide`, n-gram provider, lexical
289
+ fallback) stays live as rollback until a stage is promoted.
290
+ - The intent taxonomy and the owner-reviewed holdouts do not change here.
291
+ - Private prompts never enter this repository; public evidence holds
292
+ aggregates, hashes and per-option metrics only.
@@ -56,7 +56,7 @@ prompts written by an agent, not by the owner. VS Code terminal notifications
56
56
  - **Deduplication**: exact duplicates after whitespace normalization are dropped. The first
57
57
  occurrence is kept.
58
58
  - **Holdout leakage**: a prompt is dropped if its character-trigram Jaccard similarity to any
59
- holdout prompt is at least 0.5. The same applies if either prompt contains the other after
59
+ prompt of the tier or intent holdout is at least 0.5. The same applies if either prompt contains the other after
60
60
  normalization and the shorter one is more than 12 characters long. This matters because
61
61
  the holdout includes rewrites of local prompt shapes.
62
62
  - **Family**: the source session. Prompts from one conversation never span splits.
@@ -65,10 +65,11 @@ prompts written by an agent, not by the owner. VS Code terminal notifications
65
65
 
66
66
  ## Privacy
67
67
 
68
- - Collected prompts and labels stay on the local machine under
69
- `~/.agents/harness-everything/system-one/training/` and are **never committed**. The
70
- repository holds only the scripts, aggregate statistics and the SHA-256 of each dataset
71
- file.
68
+ - Collected prompts and labels are **never committed to this public repository**. They
69
+ live under `~/.agents/harness-everything/system-one/training/` on the owner's machine.
70
+ A curated copy, for cloud sessions, is in the owner's private repository
71
+ `dyphn1/harness-system-one-data` (`training/` is the `--data-dir`). This repository holds
72
+ only the scripts, aggregate statistics and the SHA-256 of each dataset file.
72
73
  - Email addresses and user home paths are replaced with placeholders before storage. Names
73
74
  and product terms are not reliably detectable and remain.
74
75
  - The released weights come from a 4-way byte-level classifier with about 0.7M parameters. It
@@ -88,6 +89,31 @@ prompts written by an agent, not by the owner. VS Code terminal notifications
88
89
  labels already written. Every returned index must be labeled exactly once with
89
90
  `tier1`, `tier2`, `tier3` or `null`. A malformed batch is retried once and then
90
91
  recorded as failed. It is never partially accepted.
92
+ - `--model` and `--engine codex` choose another labeler; the label rows record it.
93
+ `--batch N` changes the batch size.
94
+ - `--task intent --scores` asks for [relevance scores](system-one-intent.md#relevance-scores).
95
+ Each reply names a primary and scores every intent from 0 to 1 in steps of 0.05. A
96
+ reply is malformed when it misses an intent, leaves the scale, ties two scores above 0,
97
+ has more than eight zeros, names a primary that is not the unique top score of at
98
+ least 0.6, or has a `null` primary with an intent above 0.2. The owner chose Haiku
99
+ for this labeling.
100
+ - `--scale` scales each non-`null` scored label so its top score is 1: every score is
101
+ divided by the top score. Haiku puts most top scores at 0.75 to 0.85, and the owner
102
+ chose to read them as relative (2026-09-24). The bands and the secondary intents are
103
+ derived from the scaled scores; the row keeps the model's scores as `raw`.
104
+ - `--escalate-model M` labels a prompt a second time with model `M` when the first
105
+ label lags:
106
+ - `--escalate-margin R`: the relative margin, (top − second) / top, is below `R`.
107
+ The owner chose this lagging indicator to keep Sonnet to few prompts. `R` = 0.3
108
+ sends about 11% of Haiku's non-`null` training labels to Sonnet; it was read from
109
+ that distribution, not from the holdout. A `null` label is never escalated.
110
+ - Otherwise `--escalate-below` (default 0.8): the top score is below it. On the
111
+ training prompts this sent 69% to Sonnet, and the owner stopped it for its cost.
112
+ - The second label replaces the first; when the second run fails, the first label
113
+ stays. The label rows record the model that produced them, and a resumed run skips
114
+ prompts that were already escalated. The holdout check reports how many prompts were
115
+ escalated. `rescale --in A --out B` rewrites a label file with scaled labels (last
116
+ row per prompt).
91
117
  - **Labeler check**: the labeler labels the 215 holdout prompts once, before any training
92
118
  data is labeled. The report gives agreement with the owner's gold and the per-class recall.
93
119
  - Labeling proceeds only when agreement is at least 80%.
@@ -114,6 +140,25 @@ prompts written by an agent, not by the owner. VS Code terminal notifications
114
140
  weights and a weight decay of 1e-4.
115
141
  - The output is the `.bin` weights, the `.json` sidecar and validation scores for
116
142
  calibration.
143
+ - **Intent engine**: the same trainer with `--task intent` trains the intent stage
144
+ ([system-one-intent.md](system-one-intent.md)).
145
+ - Labels come from `labels-intent.jsonl`, and owner corrections from
146
+ `owner-overrides-intent.jsonl`. `owner-excluded.jsonl` applies to both stages. The
147
+ catalog comes from the intent holdout.
148
+ - Targets are soft. The primary gets 1.0 when there is no secondary intent. Otherwise
149
+ the primary gets 0.6 and the secondary intents share 0.4 equally. `null` maps to
150
+ `unclassified` with 1.0.
151
+ - Validation scores keep each row's secondary intents for calibration.
152
+ - `--target scores` trains on the scored labels in `labels-intent-scores.jsonl`
153
+ instead. Each intent is its own binary target with its score as the soft value, and
154
+ the loss is a per-intent binary cross-entropy. `unclassified` scores 0.6 when the
155
+ primary is `null` and 0 otherwise. An owner override without scores counts as 0.6
156
+ for its primary and 0.4 for its secondary intents. The class weights come from the
157
+ score mass of each intent.
158
+ - The artifact format is unchanged. The provider still returns the softmax over the
159
+ same logits, which ranks the intents in relevance order, so the contract's
160
+ probability vector and the calibration below stay as they are. Handing the absolute
161
+ relevance bands to the skills stage is a separate contract change.
117
162
  - The `tinyx` path below is kept for comparison. Its first checkpoint learned little beyond
118
163
  class priors.
119
164
  - **Model** (`tinyx`): a new `cua_s1` `tinyx` scorer built with `make_system`.
@@ -128,7 +173,12 @@ prompts written by an agent, not by the owner. VS Code terminal notifications
128
173
  - The epoch with the lowest validation negative log-likelihood is kept.
129
174
  - The checkpoint metadata records the dataset hashes, the config and the validation metrics.
130
175
  - **Calibration**: `minConfidence` and `minMargin` are chosen on validation to maximize
131
- coverage, subject to an accepted precision of at least 85% (the owner's advisory target). The router currently uses the
176
+ coverage, subject to an accepted precision of at least 85% (the owner's advisory target).
177
+ For the intent stage (`--task intent`), accepted precision is the mean graded agreement
178
+ of accepted rows. A predicted secondary is any other intent whose probability reaches
179
+ `secondaryThreshold`. That threshold is chosen from 0.05 to 0.50 in steps of 0.05, to
180
+ maximize micro-F1 against the labeled secondary intents, and it is chosen before the
181
+ acceptance thresholds. The router currently uses the
132
182
  Phase 0 defaults (0.9 and 0.2). Carrying calibrated thresholds into the manifest is a
133
183
  separate contract change.
134
184
 
@@ -76,6 +76,13 @@ flowchart TD
76
76
  Check -->|Pass| Handoff[Compact evidence + handoff]
77
77
  ```
78
78
 
79
+ Minor-concern batching is **not** a stage transition. Workers keep producing and
80
+ verifying the assigned artifact and include accumulated minor concerns in the next
81
+ natural report/handoff; the number of minor concerns alone never stops or returns a
82
+ worker. A material confirmed blocker may stop the current stage immediately. Repeated
83
+ matching execution failures use the separate Rule-of-3 path: zoom out, reflect, then
84
+ `RESUME` on a new approach or `ESCALATE` when a real decision is required.
85
+
79
86
  At most two full replans are allowed before unresolved blockers are escalated rather than hidden behind endless restructuring.
80
87
 
81
88
  ## 5. Completion Boundary
@@ -4,7 +4,7 @@ description: "Stage large, multi-source or multi-session tasks through a written
4
4
  license: Apache-2.0
5
5
  metadata:
6
6
  author: Miya Daniel
7
- version: 0.16.3
7
+ version: 0.30.1
8
8
  ---
9
9
 
10
10
  # Fable Mode (v3)
@@ -152,10 +152,10 @@ hasn't been confirmed present. Absence of evidence is not the finding; web
152
152
  silence is never grounds for a warning against the user's firsthand information.
153
153
  Confirm, then flag.
154
154
 
155
- **Warning threshold.** Keep a running count of minor concerns. At three
156
- accumulated (unless the briefing sets a different number), stop and surface all
157
- at once before continuing. An independently material, confirmed concern does not
158
- wait for the threshold.
155
+ **Warning batching.** Collect minor concerns and list them together in the next
156
+ natural report or handoff; their count alone MUST NOT stop, pause, or return the
157
+ worker/stage. An independently material, confirmed concern may stop the current
158
+ stage and be surfaced immediately.
159
159
 
160
160
  **Find-and-replace safety.** Anchor substring replaces on word boundaries
161
161
  (`\bword\b`, never bare `word` — a bare `edge` replace mangles `Ledger`). Prefer
@@ -41,10 +41,10 @@ Grep, diff, run it, or check the source directly. Never report a problem that
41
41
  hasn't been confirmed present. Absence of evidence is not the finding. Confirm,
42
42
  then flag.
43
43
 
44
- **Warning threshold.** Keep a running count of minor concerns. At three
45
- accumulated (unless the briefing sets a different number), stop and surface all
46
- at once before continuing. An independently material, confirmed concern does not
47
- wait for the threshold.
44
+ **Warning batching.** Collect minor concerns and list them together in the next
45
+ natural report or handoff; their count alone MUST NOT stop, pause, or return the
46
+ worker/stage. An independently material, confirmed concern may stop the current
47
+ stage and be surfaced immediately.
48
48
 
49
49
  **Find-and-replace safety.** Anchor substring replaces on word boundaries
50
50
  (`\bword\b`, never bare `word` — a bare `edge` replace mangles `Ledger`). Prefer
@@ -42,10 +42,10 @@ hasn't been confirmed present. Absence of evidence is not the finding; web
42
42
  silence is never grounds for a warning against the user's firsthand information.
43
43
  Confirm, then flag.
44
44
 
45
- **Warning threshold.** Keep a running count of minor concerns. At three
46
- accumulated (unless the briefing sets a different number), stop and surface all
47
- at once before continuing. An independently material, confirmed concern does not
48
- wait for the threshold.
45
+ **Warning batching.** Collect minor concerns and list them together in the next
46
+ natural report or handoff; their count alone MUST NOT stop, pause, or return the
47
+ worker/stage. An independently material, confirmed concern may stop the current
48
+ stage and be surfaced immediately.
49
49
 
50
50
  **Find-and-replace safety.** Anchor substring replaces on word boundaries
51
51
  (`\bword\b`, never bare `word` — a bare `edge` replace mangles `Ledger`). Prefer
@@ -7,7 +7,7 @@ description: >-
7
7
  license: Apache-2.0
8
8
  metadata:
9
9
  author: Miya Daniel
10
- version: 0.16.3
10
+ version: 0.30.1
11
11
  ---
12
12
 
13
13
  # Execution Guardrails
@@ -17,7 +17,7 @@ metadata:
17
17
  | Component | Spec |
18
18
  | :--- | :--- |
19
19
  | **Trigger / Input** | Turn raising a warning or performing search-and-replace edits. |
20
- | **Expected Output** | Verified warnings; batched caveats (3-item threshold); corruption-checked edits. |
20
+ | **Expected Output** | Verified warnings; batched caveats without count-based stops; corruption-checked edits. |
21
21
  | **State Mutations** | None — governs execution quality for file-editing turns. |
22
22
  | **Enforcement Gate** | Verify before flagging; validate edits post-write. |
23
23
 
@@ -35,9 +35,11 @@ metadata:
35
35
  ## Core Rules
36
36
 
37
37
  1. **Verify before flag** — Confirm a problem exists before reporting it. Never convert absence of evidence into a warning.
38
- 2. **Warning threshold** — Batch minor concerns; surface them together at three (default). Material concerns do not wait.
38
+ 2. **Warning batching.** Collect minor concerns and list them together in the next natural report or handoff; their count alone MUST NOT stop, pause, or return the worker/stage. An independently material, confirmed concern may stop the current stage and be surfaced immediately.
39
39
  3. **Find-and-replace safety** — Prefer structured edit tools over shell `sed`; anchor with unique context or `\bword\b` boundaries; verify file integrity post-edit. Never replace-all blindly.
40
40
 
41
+ Repeated matching execution failures remain governed by Rule-of-3 / `zoom-out`; warning batching never substitutes for that recovery path.
42
+
41
43
  These rules are always-on, even when `fable-mode`'s staged loop is not running.
42
44
 
43
45
  Deep dive: <this-skill-dir>/references/guardrail-rules.md
@@ -26,17 +26,21 @@ attempted and where it failed — not a vague appeal to difficulty.
26
26
 
27
27
  ## 2. Warning threshold
28
28
 
29
- Across any run, minor concerns accumulate that aren't worth halting on individually.
30
- Keep a running count. At the threshold — **default three, tunable if the user sets a
31
- different number** — stop and surface all of them to the user at once before continuing.
29
+ Across any run, minor concerns accumulate that are not worth interrupting execution
30
+ for individually. Keep a running count. At the threshold — **default three, tunable if
31
+ the user sets a different number** — batch them for the next natural report or handoff.
32
+ The threshold changes presentation only: it MUST NOT stop, pause, return, or otherwise
33
+ end a worker/stage by count alone.
32
34
 
33
- Rationale: three small things pointing the same direction usually mean one real problem
34
- worth a decision. Below threshold, keep working; a drip of trivial caveats is noise.
35
- At threshold, batch them — one interruption with full context beats three fragmentary
36
- ones.
35
+ Rationale: three small things pointing the same direction are easier to evaluate as one
36
+ batch. Below threshold, keep working; a drip of trivial caveats is noise. At threshold,
37
+ keep working and preserve the batch for the report instead of creating a second circuit
38
+ breaker.
37
39
 
38
40
  A concern that independently meets the verify-before-flag bar and is material on its own
39
- does not wait for the threshold. The threshold governs minor concerns only.
41
+ may stop the current stage and be surfaced immediately. Repeated matching execution
42
+ failures are governed separately by Rule-of-3 -> zoom-out -> RESUME/ESCALATE. The
43
+ warning threshold governs minor-concern reporting only.
40
44
 
41
45
  ## 3. Find-and-replace safety
42
46
 
@@ -4,7 +4,7 @@ description: "Run staged fable work on Haiku for bulk mechanical tasks with expl
4
4
  license: Apache-2.0
5
5
  metadata:
6
6
  author: Miya Daniel
7
- version: 0.16.3
7
+ version: 0.30.1
8
8
  ---
9
9
 
10
10
  # Fable Mode — Haiku
@@ -4,7 +4,7 @@ description: "Run staged fable orchestration on Opus for cross-stage synthesis,
4
4
  license: Apache-2.0
5
5
  metadata:
6
6
  author: Miya Daniel
7
- version: 0.16.3
7
+ version: 0.30.1
8
8
  ---
9
9
 
10
10
  # Fable Mode — Opus
@@ -4,7 +4,7 @@ description: "Run staged fable work on Sonnet for non-trivial implementation, re
4
4
  license: Apache-2.0
5
5
  metadata:
6
6
  author: Miya Daniel
7
- version: 0.16.3
7
+ version: 0.30.1
8
8
  ---
9
9
 
10
10
  # Fable Mode — Sonnet
@@ -4,7 +4,7 @@ description: "Route software/project work through Harness: classify tier/topolog
4
4
  license: Apache-2.0
5
5
  metadata:
6
6
  author: Miya Daniel
7
- version: 0.29.0
7
+ version: 0.30.0
8
8
  ---
9
9
 
10
10
  # Harness Everything