harness-everything 0.29.0 → 0.30.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/.claude-plugin/marketplace.json +1 -1
- package/.claude-plugin/plugin.json +1 -1
- package/AGENTS.md +1 -0
- package/CHANGELOG.md +42 -0
- package/PRIVACY.md +9 -0
- package/benchmarks/fixtures/system-one-intent-holdout-draft.json +2228 -0
- package/benchmarks/fixtures/system-one-intent-holdout.json +16509 -0
- package/benchmarks/fixtures/system-one-intent-reviews.json +2723 -0
- package/benchmarks/results/system-one/intent-ngram-v1-2026-09-24/README.md +64 -0
- package/benchmarks/results/system-one/intent-ngram-v1-2026-09-24/report.json +15177 -0
- package/benchmarks/results/system-one/labeler-holdout-check-intent-2026-09-24/report.json +2976 -0
- package/benchmarks/results/system-one/labeler-holdout-check-intent-scores-haiku-2026-09-24/report-005-steps.json +6125 -0
- package/benchmarks/results/system-one/labeler-holdout-check-intent-scores-haiku-2026-09-24/report-haiku-sonnet-escalation.json +6344 -0
- package/benchmarks/results/system-one/labeler-holdout-check-intent-scores-haiku-2026-09-24/report-scaled-margin-escalation.json +6206 -0
- package/benchmarks/results/system-one/labeler-holdout-check-intent-scores-haiku-2026-09-24/report.json +6103 -0
- package/benchmarks/results/system-one/laya-ft-2026-09-25/README.md +83 -0
- package/benchmarks/results/system-one/laya-ft-2026-09-25/report.json +71 -0
- package/benchmarks/results/system-one/laya-ft-2026-09-25/shadow-report.json +44 -0
- package/benchmarks/results/system-one/laya-zs-2026-09-25/README.md +68 -0
- package/benchmarks/results/system-one/laya-zs-2026-09-25/report.json +173 -0
- package/ci/fable-mode-test.js +16 -0
- package/ci/mechanism-34-observations.test.js +550 -0
- package/ci/mechanism-system-one-collect.test.js +16 -0
- package/ci/mechanism-system-one-intent.test.js +406 -0
- package/ci/mechanism-system-one-laya.test.js +45 -0
- package/ci/mechanism-system-one-relevance.test.js +112 -0
- package/ci/mechanism-system-one-resident.test.js +16 -0
- package/ci/system-one-laya-calibrate-test.py +65 -0
- package/ci/system-one-laya-export-test.py +155 -0
- package/ci/system-one-laya-label-test.py +201 -0
- package/ci/system-one-laya-perintent-test.py +60 -0
- package/ci/system-one-laya-provider-test.py +115 -0
- package/ci/system-one-laya-train-test.py +93 -0
- package/docs/routing.md +25 -0
- package/docs/system-one-intent-gates.md +102 -0
- package/docs/system-one-intent.md +173 -0
- package/docs/system-one-laya-training.md +38 -0
- package/docs/system-one-observations.md +206 -0
- package/docs/system-one-routing.md +25 -10
- package/docs/system-one-suggestion-gates.md +292 -0
- package/docs/system-one-training.md +56 -6
- package/docs/workflows/fable-mode.md +7 -0
- package/fable-mode/SKILL.md +1 -1
- package/fable-mode/agents/fable-orchestrator.md +4 -4
- package/fable-mode/agents/fable-worker-haiku.md +4 -4
- package/fable-mode/agents/fable-worker-sonnet.md +4 -4
- package/fable-mode/execution-guardrails/SKILL.md +5 -3
- package/fable-mode/execution-guardrails/references/guardrail-rules.md +12 -8
- package/fable-mode/fable-haiku/SKILL.md +1 -1
- package/fable-mode/fable-opus/SKILL.md +1 -1
- package/fable-mode/fable-sonnet/SKILL.md +1 -1
- package/harness-everything/SKILL.md +1 -1
- package/harness-everything/scripts/kernel-router.js +10 -0
- package/harness-everything/scripts/system-one/catalogs.js +35 -0
- package/harness-everything/scripts/system-one/contract.js +45 -1
- package/harness-everything/scripts/system-one/evaluate.js +115 -19
- package/harness-everything/scripts/system-one/intent.js +22 -0
- package/harness-everything/scripts/system-one/laya_adapter.py +141 -0
- package/harness-everything/scripts/system-one/provider.js +10 -3
- package/harness-everything/scripts/system-one/resident.js +8 -3
- package/harness-everything/scripts/system-one/router.js +1 -1
- package/hooks/hooks.json +40 -0
- package/hooks/scripts/codex-user-prompt.js +39 -0
- package/hooks/scripts/lib/observations.js +581 -0
- package/hooks/scripts/observation-hook.js +25 -0
- package/opencode-plugin/plugin.json +1 -1
- package/package.json +1 -1
- package/plugins/harness-everything/.codex-plugin/plugin.json +1 -1
- package/plugins/harness-everything/hooks/hooks.json +35 -2
- package/plugins/harness-everything/hooks/scripts/codex-user-prompt.js +39 -0
- package/plugins/harness-everything/hooks/scripts/lib/observations.js +581 -0
- package/plugins/harness-everything/hooks/scripts/observation-hook.js +25 -0
- package/plugins/harness-everything/plugin.json +1 -1
- package/plugins/harness-everything/skills/fable-mode/SKILL.md +1 -1
- package/plugins/harness-everything/skills/fable-mode/agents/fable-orchestrator.md +4 -4
- package/plugins/harness-everything/skills/fable-mode/agents/fable-worker-haiku.md +4 -4
- package/plugins/harness-everything/skills/fable-mode/agents/fable-worker-sonnet.md +4 -4
- package/plugins/harness-everything/skills/fable-mode/execution-guardrails/SKILL.md +5 -3
- package/plugins/harness-everything/skills/fable-mode/execution-guardrails/references/guardrail-rules.md +12 -8
- package/plugins/harness-everything/skills/fable-mode/fable-haiku/SKILL.md +1 -1
- package/plugins/harness-everything/skills/fable-mode/fable-opus/SKILL.md +1 -1
- package/plugins/harness-everything/skills/fable-mode/fable-sonnet/SKILL.md +1 -1
- package/plugins/harness-everything/skills/harness-everything/SKILL.md +1 -1
- package/plugins/harness-everything/skills/harness-everything/scripts/kernel-router.js +10 -0
- package/plugins/harness-everything/skills/harness-everything/scripts/system-one/catalogs.js +35 -0
- package/plugins/harness-everything/skills/harness-everything/scripts/system-one/contract.js +45 -1
- package/plugins/harness-everything/skills/harness-everything/scripts/system-one/evaluate.js +115 -19
- package/plugins/harness-everything/skills/harness-everything/scripts/system-one/intent.js +22 -0
- package/plugins/harness-everything/skills/harness-everything/scripts/system-one/laya_adapter.py +141 -0
- package/plugins/harness-everything/skills/harness-everything/scripts/system-one/provider.js +10 -3
- package/plugins/harness-everything/skills/harness-everything/scripts/system-one/resident.js +8 -3
- package/plugins/harness-everything/skills/harness-everything/scripts/system-one/router.js +1 -1
- package/sbom.cdx.json +7 -7
- package/scripts/evaluate-system-one.js +17 -5
- package/scripts/system-one-calibrate-laya.py +165 -0
- package/scripts/system-one-calibrate.js +42 -13
- package/scripts/system-one-collect.js +7 -3
- package/scripts/system-one-compare-laya.py +156 -0
- package/scripts/system-one-corpus.js +22 -15
- package/scripts/system-one-eval-laya.js +112 -0
- package/scripts/system-one-export-laya.py +197 -0
- package/scripts/system-one-label-laya.py +243 -0
- package/scripts/system-one-label.js +163 -37
- package/scripts/system-one-observations-export.js +106 -0
- package/scripts/system-one-observations-review.js +86 -0
- package/scripts/system-one-perintent-laya.py +134 -0
- package/scripts/system-one-train-laya.py +344 -0
- package/scripts/system-one-train-ngram.py +66 -14
|
@@ -42,10 +42,16 @@ The manifest is version 1 with `python` (executable), `checkpoint` (absolute
|
|
|
42
42
|
`.safetensors` path), `weightsSha256`, `configSha256` (matching `.json` sidecar),
|
|
43
43
|
`modelId`, `revision`, `domain`, and optional `timeoutMs` (1–10000, default 2000).
|
|
44
44
|
All fields are required except the timeout; unknown fields are rejected. An
|
|
45
|
-
optional `acceptance` object holds
|
|
46
|
-
`minMargin` (0–0.9).
|
|
45
|
+
optional `acceptance` object holds `minConfidence` (0.5–0.99) and
|
|
46
|
+
`minMargin` (0–0.9). It may also hold `secondaryThreshold` (0.05–0.5), the
|
|
47
|
+
probability at which another intent counts as a secondary intent; that key is
|
|
48
|
+
valid only for an intent checkpoint ([system-one-intent.md](system-one-intent.md#scoring)).
|
|
49
|
+
These are the thresholds calibrated on validation for this
|
|
47
50
|
checkpoint. A scored result carries them, and the router and the evaluator pass
|
|
48
|
-
|
|
51
|
+
the first two to `decide`. Without the field the Phase 0 defaults apply. For an
|
|
52
|
+
intent corpus the evaluator also reports graded accepted precision, exact accepted
|
|
53
|
+
precision and family consistency. The `acceptedPrecision` gate uses the graded
|
|
54
|
+
value. Hashes
|
|
49
55
|
must be lowercase SHA-256. Revision and domain are artifact-owner declarations,
|
|
50
56
|
not independently verified quality claims. The known `cua-ai/cua-s1-forms` ID
|
|
51
57
|
must use domain `forms-v1`. No pickle format or remote URL is accepted.
|
|
@@ -349,23 +355,32 @@ small model only has to separate a few options at a time.
|
|
|
349
355
|
|
|
350
356
|
1. **Tier**: `tier1`, `tier2`, `tier3` or `unclassified`, following
|
|
351
357
|
[system-one-corpus.md](system-one-corpus.md).
|
|
352
|
-
2. **Intent**: the kind of work the prompt asks for, drawn from
|
|
353
|
-
catalog (
|
|
354
|
-
|
|
355
|
-
|
|
356
|
-
|
|
358
|
+
2. **Intent**: the kind of work the prompt asks for, drawn from the owner's
|
|
359
|
+
fixed catalog in [system-one-intent.md](system-one-intent.md): `explain`,
|
|
360
|
+
`discuss`, `git`, `fix`, `edit`, `feature`, `refactor`, `review`, `test`,
|
|
361
|
+
`docs`, `plan`, `investigate` or `unclassified`. A prompt has one primary
|
|
362
|
+
intent and may have secondary intents.
|
|
357
363
|
3. **Skills**: which canonical skills fit the prompt, scored against skill
|
|
358
364
|
descriptions. Only skills above the stage's calibrated threshold are
|
|
359
365
|
suggested, and they are suggestions, never required reads.
|
|
360
366
|
|
|
361
|
-
Each stage has its own gates, and a stage ships only when it passes them.
|
|
362
|
-
|
|
367
|
+
Each stage has its own gates, and a stage ships only when it passes them.
|
|
368
|
+
The relevance-native plan in
|
|
369
|
+
[system-one-suggestion-gates.md](system-one-suggestion-gates.md#readout-per-stage)
|
|
370
|
+
adds a validity stage first and runs intent before tier, because tier3 is
|
|
371
|
+
composed from the intent scores. Tier was trained first because the
|
|
372
|
+
reviewed holdout exists for it. Later stages reuse the
|
|
363
373
|
same collector, labeler, trainer and evaluator with a different catalog.
|
|
364
374
|
Suggestions are advisory: explicit workflow requests, action gates, memory
|
|
365
375
|
ownership, the Rule of 3 and deterministic policy always take precedence.
|
|
366
376
|
|
|
367
377
|
## Rollout gates
|
|
368
378
|
|
|
379
|
+
Model-quality gates for relevance-native candidates are defined in
|
|
380
|
+
[system-one-suggestion-gates.md](system-one-suggestion-gates.md), which
|
|
381
|
+
supersedes the precision/coverage/latency targets below. The holdout size,
|
|
382
|
+
provenance, repeatability and advisory rules below still apply.
|
|
383
|
+
|
|
369
384
|
Phase 4 requires a separately reviewed, family-disjoint holdout of at least 200
|
|
370
385
|
cases, at least 50 each in English and Traditional Chinese. System One output is
|
|
371
386
|
advisory: it informs the agent and suggests skills. It never grants or removes
|
|
@@ -0,0 +1,292 @@
|
|
|
1
|
+
# System One suggestion gates
|
|
2
|
+
|
|
3
|
+
Status: **proposed, pending owner review**. Once the owner approves it, this
|
|
4
|
+
document is frozen for the next candidate of each stage. Decided in the
|
|
5
|
+
[#233 design correction](https://github.com/dyphn1/Harness-everything/issues/233#issuecomment-5842359628)
|
|
6
|
+
(one event, matched against every option); research evidence is in
|
|
7
|
+
[#255](https://github.com/dyphn1/Harness-everything/issues/255).
|
|
8
|
+
|
|
9
|
+
This document supersedes the table in
|
|
10
|
+
[system-one-intent-gates.md](system-one-intent-gates.md). That table stays
|
|
11
|
+
as the historical contract for the runs made under it. It also supersedes
|
|
12
|
+
the model-quality part of
|
|
13
|
+
[system-one-routing.md#rollout-gates](system-one-routing.md#rollout-gates).
|
|
14
|
+
`evaluate.relevanceGates` still implements the superseded table. The code
|
|
15
|
+
change follows this document (docs → RED → GREEN) in its own commit.
|
|
16
|
+
|
|
17
|
+
## Principle
|
|
18
|
+
|
|
19
|
+
1. **One event, scored against every option.** For a prompt, the scorer
|
|
20
|
+
returns an independent relevance `p_i ∈ [0, 1]` for every option of the
|
|
21
|
+
stage catalog. Scores do not sum to 1 and are never renormalized into a
|
|
22
|
+
simplex. With CUA-S1 this is the per-option logit passed through a
|
|
23
|
+
sigmoid. Upstream `tinyx` already encodes each option separately and
|
|
24
|
+
scores it against the context, so one forward pass scores every option.
|
|
25
|
+
2. **Suggestions, not verdicts.** Output is advice to the agent. A wrong
|
|
26
|
+
suggestion costs little; a confident wrong suggestion, or a long list the
|
|
27
|
+
agent must read, costs trust and tokens. Gates therefore measure
|
|
28
|
+
*trustworthy confidence* and *suggestion coverage under a noise cap*,
|
|
29
|
+
not exact accuracy.
|
|
30
|
+
3. **Deterministic controls stay outside learned scoring.** Tier structural
|
|
31
|
+
floor, explicit workflow requests, action/security approval, scope and
|
|
32
|
+
memory ownership and the Rule of 3 are unchanged. Abstention always falls
|
|
33
|
+
back to the lexical path.
|
|
34
|
+
|
|
35
|
+
## Readout per stage
|
|
36
|
+
|
|
37
|
+
All stages share the scoring and the calibration. Only the readout differs.
|
|
38
|
+
|
|
39
|
+
| Stage | Catalog | Readout | Cap |
|
|
40
|
+
| --- | --- | --- | --- |
|
|
41
|
+
| Validity | `actionable`, `invalid` | `invalid` at or above its threshold → stop: no tier, workflow, intent or skill suggestion; the host agent decides | — |
|
|
42
|
+
| Intent | the 12 intents of [system-one-intent.md](system-one-intent.md) | set: every intent at or above its threshold, highest `p` first, truncated at the cap | 3 |
|
|
43
|
+
| Tier | `tier1`, `tier2`, `tier3` | composed (see [Tier composition](#tier-composition)): tier3 when `feature` or `refactor` reaches its intent threshold; otherwise the tier scorer's pick; final tier = max(that, structural floor); none → abstain | 1 |
|
|
44
|
+
| Workflow | the router's strategies (`direct-single`, `iterative-single`, `fable-staged`, `fable-parallel`, `fable-multi-agent-workspace`) | pick one, same rule; an explicit workflow request always wins | 1 |
|
|
45
|
+
| Skills | canonical skills, scored against their descriptions | set, same rule | 3 |
|
|
46
|
+
|
|
47
|
+
The cap of 3 for set stages comes from AGENTS.md rule 10: a suggested skill
|
|
48
|
+
must be read before it can be dismissed, so each extra suggestion has a
|
|
49
|
+
reading cost. The cap is part of the readout, so noise is bounded by
|
|
50
|
+
construction; the gates then judge what survives the cap.
|
|
51
|
+
|
|
52
|
+
Stages run in table order: validity, intent, tier, workflow, skills. Tier
|
|
53
|
+
reads the intent scores, so intent runs before it.
|
|
54
|
+
|
|
55
|
+
### Tier composition
|
|
56
|
+
|
|
57
|
+
Tier rules v2 (owner decision, 2026-09-26): tier3 covers new features,
|
|
58
|
+
refactors, changes to definitions, and cross-repository or cross-component
|
|
59
|
+
work, whatever the width of the change; tier1 is a clear bounded action
|
|
60
|
+
(Git, status, lookup, direct answer); tier2 is a bounded modification.
|
|
61
|
+
|
|
62
|
+
Whether a prompt asks for a feature or a refactor is an intent question,
|
|
63
|
+
and the intent stage already scores it. The tier stage therefore does not
|
|
64
|
+
learn that part again:
|
|
65
|
+
|
|
66
|
+
1. **Intent part.** tier3 when the calibrated score of `feature` or
|
|
67
|
+
`refactor` is at or above that intent's threshold (either one fires; not
|
|
68
|
+
"is the top intent", which a small intent scorer rarely satisfies).
|
|
69
|
+
2. **Scope part.** The deterministic structural floor (cross-repository,
|
|
70
|
+
cross-component, macro scope) sets tier3 as before.
|
|
71
|
+
3. **Tier scorer.** Otherwise the tier scorer picks, trained on the
|
|
72
|
+
breadth-based labels, so it separates tier1 from tier2 and keeps the
|
|
73
|
+
breadth-based tier3.
|
|
74
|
+
|
|
75
|
+
Over-reading `feature`/`refactor` moves a prompt one tier up, which the
|
|
76
|
+
gates count as acceptable; missing it moves it down, which they count as
|
|
77
|
+
under-tier. The composition is therefore tuned for intent recall on those
|
|
78
|
+
two intents.
|
|
79
|
+
|
|
80
|
+
Evidence (#265, validation, five seeds): composed with the CUA-S1 intent
|
|
81
|
+
scorer, acceptable precision 0.82 and under-tier 0.13, against 0.77 and
|
|
82
|
+
0.22 for a tier scorer trained directly on rules-v2 labels.
|
|
83
|
+
|
|
84
|
+
**Holdout.** The owner decided not to re-review the tier holdout under
|
|
85
|
+
rules v2. Its gold follows the older rules, and rules v2 only moves labels
|
|
86
|
+
up (tier1 or tier2 to tier3). So underRate on the old gold stays valid, and
|
|
87
|
+
a tier3 pick for tier2 gold counts as acceptable. A tier3 pick for a
|
|
88
|
+
tier1-gold feature prompt counts as an error, which makes
|
|
89
|
+
acceptablePrecision slightly pessimistic; the count of such cases is
|
|
90
|
+
reported beside it.
|
|
91
|
+
|
|
92
|
+
`unclassified`/`null` is not an option in the new readout. It is the
|
|
93
|
+
abstain outcome: no option reaches its threshold.
|
|
94
|
+
|
|
95
|
+
**The scorer only sees the current prompt.** It never receives earlier
|
|
96
|
+
turns. A prompt whose text alone carries no actionable request is
|
|
97
|
+
`invalid`: System One makes no suggestion, and the host agent, which has
|
|
98
|
+
the conversation, decides. Validity runs first; the later stages run only
|
|
99
|
+
on `actionable` prompts.
|
|
100
|
+
|
|
101
|
+
## Calibration and thresholds
|
|
102
|
+
|
|
103
|
+
- **Calibration.** A monotone per-option map (isotonic or Platt, chosen and
|
|
104
|
+
recorded before fitting) from raw `p` to calibrated `p`, fitted on
|
|
105
|
+
validation only. Needed because a model trained on soft teacher scores
|
|
106
|
+
outputs a *relevance degree* (0.5 = "secondary relevance"), not a
|
|
107
|
+
probability of being relevant. Gates read calibrated `p`.
|
|
108
|
+
- **Suggest threshold `τ_i`.** One per option, fitted on validation
|
|
109
|
+
(max-F1 against the validation truth), 2-fold CV reported.
|
|
110
|
+
- **Confident band.** Calibrated `p ≥ 0.8`. A confident suggestion is a
|
|
111
|
+
claim that at least 8 in 10 such suggestions are relevant.
|
|
112
|
+
- Calibration maps, thresholds, cap, band and this document are frozen
|
|
113
|
+
before a holdout run. Nothing is refit after it.
|
|
114
|
+
|
|
115
|
+
## Truth
|
|
116
|
+
|
|
117
|
+
| Split | Relevant set | Primary |
|
|
118
|
+
| --- | --- | --- |
|
|
119
|
+
| Train / validation | teacher score ≥ 0.4, after owner intent overrides and exclusions | teacher primary |
|
|
120
|
+
| Intent holdout | owner `gold ∪ secondary` | owner `gold` |
|
|
121
|
+
| Tier holdout | owner `gold` tier; `null` means abstention is correct | — |
|
|
122
|
+
|
|
123
|
+
### Invalid prompts
|
|
124
|
+
|
|
125
|
+
Some prompts cannot be judged from their text: a go-ahead (`go`, `好`,
|
|
126
|
+
`完成了`), an option pick, a change verb with no object (`修正一下`),
|
|
127
|
+
feedback on earlier work (`還是一樣的錯誤`), a pointer to earlier content
|
|
128
|
+
(`請依照需求實作`), a paste with no request, or chatter. The small model
|
|
129
|
+
does not try to recover the missing context. It marks them `invalid` and
|
|
130
|
+
hands them to the host agent (the external LLM), which sees the whole
|
|
131
|
+
conversation. Deterministic controls (action gates, scope and memory
|
|
132
|
+
ownership, the Rule of 3) apply unchanged. The noise-gate experiment (#265)
|
|
133
|
+
showed that tier labels given to such prompts from session context were
|
|
134
|
+
what kept a text-only model unsure.
|
|
135
|
+
|
|
136
|
+
- **Definition.** A holdout case is invalid when the frozen rules
|
|
137
|
+
(`rule_v2` in `scripts/system-one-noise-experiment.py`, #265) match its
|
|
138
|
+
text or its owner gold is `null`. The owner reviews that list before the
|
|
139
|
+
holdout run; the list is then frozen with the rest of step 4 below.
|
|
140
|
+
- **Scoring.** On an invalid case, the model is correct when it returns
|
|
141
|
+
`invalid` (or abstains); any tier, intent or skill suggestion is an error
|
|
142
|
+
for that case. Invalid cases are excluded from the later stages' metrics
|
|
143
|
+
and reported as invalid recall. On actionable cases, returning `invalid`
|
|
144
|
+
is reported as invalid false-positive rate.
|
|
145
|
+
- **Size.** The rules mark 9 of 215 tier-holdout cases (6 `null`, 2 `tier1`,
|
|
146
|
+
1 `tier2`) and 3 of 222 intent-holdout cases; the tier holdout has 20
|
|
147
|
+
`null` cases in all. The holdouts were built with most continuations
|
|
148
|
+
removed, so the validity gate is read on validation until a reviewed
|
|
149
|
+
invalid set exists.
|
|
150
|
+
|
|
151
|
+
## Metrics
|
|
152
|
+
|
|
153
|
+
Set stages (intent, skills), over actionable prompts:
|
|
154
|
+
|
|
155
|
+
| Metric | Definition |
|
|
156
|
+
| --- | --- |
|
|
157
|
+
| suggestionCoverage | share of prompts with at least one suggestion |
|
|
158
|
+
| confidentCoverage | share of prompts with at least one confident suggestion |
|
|
159
|
+
| primaryHit | share of prompts whose primary is among the suggestions |
|
|
160
|
+
| suggestionPrecision | relevant suggestions / all suggestions |
|
|
161
|
+
| confidentPrecision | relevant confident suggestions / all confident suggestions |
|
|
162
|
+
| ECE | expected calibration error over all (prompt, option) pairs, 10 equal-width bins of calibrated `p` against set membership |
|
|
163
|
+
| Brier | mean squared error of calibrated `p` against set membership |
|
|
164
|
+
| per-option precision / recall / support | support = holdout prompts where the option is relevant |
|
|
165
|
+
| cardinality | mean and p95 suggestions per prompt, share of prompts at the cap |
|
|
166
|
+
| recall | relevant suggestions / relevant options (reported) |
|
|
167
|
+
|
|
168
|
+
ECE alone rewards a timid model that never says anything confident; it is
|
|
169
|
+
gated together with confidentCoverage, never alone.
|
|
170
|
+
|
|
171
|
+
Pick stages (tier, workflow):
|
|
172
|
+
|
|
173
|
+
| Metric | Definition |
|
|
174
|
+
| --- | --- |
|
|
175
|
+
| coverage | share of actionable prompts that get a pick |
|
|
176
|
+
| acceptablePrecision | picks equal to gold or one tier above it, among picks |
|
|
177
|
+
| underRate | picks below gold, among picks |
|
|
178
|
+
| exactPrecision | picks equal to gold, among picks (reported) |
|
|
179
|
+
| ECE / Brier | as for set stages, one-vs-rest per option |
|
|
180
|
+
|
|
181
|
+
One tier above gold is acceptable because over-routing adds process;
|
|
182
|
+
under-routing skips obligations, so it has its own hard gate.
|
|
183
|
+
|
|
184
|
+
Every stage also reports family consistency (same family, same top
|
|
185
|
+
option), repeatability, latency, artifact size and policy evidence.
|
|
186
|
+
|
|
187
|
+
## Gates
|
|
188
|
+
|
|
189
|
+
### Intent (holdout)
|
|
190
|
+
|
|
191
|
+
Anchors are validation-proxy measurements, not holdout results: the
|
|
192
|
+
fine-tuned Laya reference, capped at 3 with in-sample thresholds, reaches
|
|
193
|
+
coverage 0.97, primary hit 0.75, precision 0.63, raw-`p ≥ 0.6` precision
|
|
194
|
+
0.85 and ECE 0.06. Holdout truth is stricter, so bars sit below it.
|
|
195
|
+
|
|
196
|
+
| Gate | Bar |
|
|
197
|
+
| --- | --- |
|
|
198
|
+
| suggestionCoverage | ≥ 0.90 |
|
|
199
|
+
| primaryHit | ≥ 0.65 |
|
|
200
|
+
| suggestionPrecision | ≥ 0.60 |
|
|
201
|
+
| confidentCoverage | ≥ 0.40 |
|
|
202
|
+
| confidentPrecision | ≥ 0.75 (the 0.8 claim minus about two standard errors at about 250 confident suggestions) |
|
|
203
|
+
| ECE | ≤ 0.10 |
|
|
204
|
+
| per-intent precision, support ≥ 20 | ≥ 0.40 |
|
|
205
|
+
| family consistency (top suggestion) | ≥ 0.70 (owner gold 0.80) |
|
|
206
|
+
|
|
207
|
+
### Tier (holdout)
|
|
208
|
+
|
|
209
|
+
| Gate | Bar |
|
|
210
|
+
| --- | --- |
|
|
211
|
+
| acceptablePrecision | ≥ 0.85 (the owner's advisory bar, now counting one tier above gold) |
|
|
212
|
+
| underRate | ≤ 0.05 |
|
|
213
|
+
| coverage | ≥ 0.60 |
|
|
214
|
+
| ECE | ≤ 0.10 |
|
|
215
|
+
| invalid recall (invalid cases) | reported; ≥ 0.80 on validation invalid rows |
|
|
216
|
+
| invalid false-positive rate (actionable cases) | ≤ 0.05 |
|
|
217
|
+
|
|
218
|
+
Tier bars have no validation anchor from a relevance-native tier model
|
|
219
|
+
yet. The first tier candidate reports its validation numbers first, and
|
|
220
|
+
these bars are confirmed or amended in this document **before** its
|
|
221
|
+
holdout run, never after.
|
|
222
|
+
|
|
223
|
+
### Workflow and skills
|
|
224
|
+
|
|
225
|
+
No reviewed holdout exists for these stages, so they cannot be promoted.
|
|
226
|
+
Before any gate applies, each needs:
|
|
227
|
+
|
|
228
|
+
- a fixed catalog document (skills: option text written for scoring; the
|
|
229
|
+
current 96-byte option limit truncates skill descriptions, so the
|
|
230
|
+
checkpoint's `option_tokens` must be sized for them);
|
|
231
|
+
- an owner-reviewed, family-disjoint holdout of at least 200 cases, at
|
|
232
|
+
least 50 each in English and Traditional Chinese;
|
|
233
|
+
- the paired lexical baseline on that holdout.
|
|
234
|
+
|
|
235
|
+
Their gates take the shape of tier (workflow) and intent (skills). A
|
|
236
|
+
learned stage must also be no worse than the lexical baseline on
|
|
237
|
+
primaryHit/suggestionPrecision (skills) or acceptablePrecision (workflow)
|
|
238
|
+
in the same paired run.
|
|
239
|
+
|
|
240
|
+
### Every stage
|
|
241
|
+
|
|
242
|
+
| Gate | Bar |
|
|
243
|
+
| --- | --- |
|
|
244
|
+
| repeatability | identical scores and suggestions across two runs |
|
|
245
|
+
| warm p95 | ≤ 250 ms, with the host record ([latency note](system-one-intent-gates.md#latency-note)) |
|
|
246
|
+
| deployable artifact | CPU only, checkpoint ≤ 10 MB, installed by `npm run system-one:install` with pinned hashes |
|
|
247
|
+
| structural controls | unchanged, rollback path live |
|
|
248
|
+
| policyEvidence / liveHostEvidence | reported; required before the default changes from `shadow` |
|
|
249
|
+
|
|
250
|
+
The artifact gate encodes the #255 decision: Laya is a teacher and a
|
|
251
|
+
ceiling reference, never a production candidate.
|
|
252
|
+
|
|
253
|
+
## Evaluation order
|
|
254
|
+
|
|
255
|
+
1. Train on train rows only.
|
|
256
|
+
2. Fit calibration and thresholds on validation (2-fold CV reported).
|
|
257
|
+
3. **Qualify on validation.** Report every metric above against the
|
|
258
|
+
validation truth. A candidate may spend a holdout run only if it meets
|
|
259
|
+
the holdout bars on validation *and* passes the deployable-artifact
|
|
260
|
+
gate. Otherwise it stays a validation result.
|
|
261
|
+
4. Freeze the checkpoint hash, calibration, thresholds, evaluator
|
|
262
|
+
revision and this document.
|
|
263
|
+
5. Score the holdout once, with two runs for repeatability.
|
|
264
|
+
6. A fail is final for that candidate. A new candidate restarts at step 1
|
|
265
|
+
with a recorded reason.
|
|
266
|
+
|
|
267
|
+
## Holdout budget
|
|
268
|
+
|
|
269
|
+
Every holdout run spends the holdout a little: each published result makes
|
|
270
|
+
later choices less independent. Each run is recorded in the evidence
|
|
271
|
+
directory with candidate hash, date and aggregate result.
|
|
272
|
+
|
|
273
|
+
- Intent holdout, spent before this document: n-gram intent v1,
|
|
274
|
+
fine-tuned Laya (single-winner), fine-tuned Laya (relevance shadow).
|
|
275
|
+
- Tier holdout, spent before this document: tinyx v1 and the n-gram
|
|
276
|
+
versions reported in #233, plus the lexical baseline.
|
|
277
|
+
|
|
278
|
+
Under this document, each holdout allows **one** further candidate run.
|
|
279
|
+
Before a second run, the owner adds at least 50 freshly reviewed cases (at
|
|
280
|
+
least 20 per language) that no earlier candidate or threshold saw, and the
|
|
281
|
+
gates are read on the enlarged holdout.
|
|
282
|
+
|
|
283
|
+
Earlier holdout results are historical. They are never re-scored under
|
|
284
|
+
these gates to claim a pass.
|
|
285
|
+
|
|
286
|
+
## Non-goals
|
|
287
|
+
|
|
288
|
+
- The single-winner path (`contract.decide`, n-gram provider, lexical
|
|
289
|
+
fallback) stays live as rollback until a stage is promoted.
|
|
290
|
+
- The intent taxonomy and the owner-reviewed holdouts do not change here.
|
|
291
|
+
- Private prompts never enter this repository; public evidence holds
|
|
292
|
+
aggregates, hashes and per-option metrics only.
|
|
@@ -56,7 +56,7 @@ prompts written by an agent, not by the owner. VS Code terminal notifications
|
|
|
56
56
|
- **Deduplication**: exact duplicates after whitespace normalization are dropped. The first
|
|
57
57
|
occurrence is kept.
|
|
58
58
|
- **Holdout leakage**: a prompt is dropped if its character-trigram Jaccard similarity to any
|
|
59
|
-
holdout
|
|
59
|
+
prompt of the tier or intent holdout is at least 0.5. The same applies if either prompt contains the other after
|
|
60
60
|
normalization and the shorter one is more than 12 characters long. This matters because
|
|
61
61
|
the holdout includes rewrites of local prompt shapes.
|
|
62
62
|
- **Family**: the source session. Prompts from one conversation never span splits.
|
|
@@ -65,10 +65,11 @@ prompts written by an agent, not by the owner. VS Code terminal notifications
|
|
|
65
65
|
|
|
66
66
|
## Privacy
|
|
67
67
|
|
|
68
|
-
- Collected prompts and labels
|
|
69
|
-
`~/.agents/harness-everything/system-one/training/`
|
|
70
|
-
|
|
71
|
-
|
|
68
|
+
- Collected prompts and labels are **never committed to this public repository**. They
|
|
69
|
+
live under `~/.agents/harness-everything/system-one/training/` on the owner's machine.
|
|
70
|
+
A curated copy, for cloud sessions, is in the owner's private repository
|
|
71
|
+
`dyphn1/harness-system-one-data` (`training/` is the `--data-dir`). This repository holds
|
|
72
|
+
only the scripts, aggregate statistics and the SHA-256 of each dataset file.
|
|
72
73
|
- Email addresses and user home paths are replaced with placeholders before storage. Names
|
|
73
74
|
and product terms are not reliably detectable and remain.
|
|
74
75
|
- The released weights come from a 4-way byte-level classifier with about 0.7M parameters. It
|
|
@@ -88,6 +89,31 @@ prompts written by an agent, not by the owner. VS Code terminal notifications
|
|
|
88
89
|
labels already written. Every returned index must be labeled exactly once with
|
|
89
90
|
`tier1`, `tier2`, `tier3` or `null`. A malformed batch is retried once and then
|
|
90
91
|
recorded as failed. It is never partially accepted.
|
|
92
|
+
- `--model` and `--engine codex` choose another labeler; the label rows record it.
|
|
93
|
+
`--batch N` changes the batch size.
|
|
94
|
+
- `--task intent --scores` asks for [relevance scores](system-one-intent.md#relevance-scores).
|
|
95
|
+
Each reply names a primary and scores every intent from 0 to 1 in steps of 0.05. A
|
|
96
|
+
reply is malformed when it misses an intent, leaves the scale, ties two scores above 0,
|
|
97
|
+
has more than eight zeros, names a primary that is not the unique top score of at
|
|
98
|
+
least 0.6, or has a `null` primary with an intent above 0.2. The owner chose Haiku
|
|
99
|
+
for this labeling.
|
|
100
|
+
- `--scale` scales each non-`null` scored label so its top score is 1: every score is
|
|
101
|
+
divided by the top score. Haiku puts most top scores at 0.75 to 0.85, and the owner
|
|
102
|
+
chose to read them as relative (2026-09-24). The bands and the secondary intents are
|
|
103
|
+
derived from the scaled scores; the row keeps the model's scores as `raw`.
|
|
104
|
+
- `--escalate-model M` labels a prompt a second time with model `M` when the first
|
|
105
|
+
label lags:
|
|
106
|
+
- `--escalate-margin R`: the relative margin, (top − second) / top, is below `R`.
|
|
107
|
+
The owner chose this lagging indicator to keep Sonnet to few prompts. `R` = 0.3
|
|
108
|
+
sends about 11% of Haiku's non-`null` training labels to Sonnet; it was read from
|
|
109
|
+
that distribution, not from the holdout. A `null` label is never escalated.
|
|
110
|
+
- Otherwise `--escalate-below` (default 0.8): the top score is below it. On the
|
|
111
|
+
training prompts this sent 69% to Sonnet, and the owner stopped it for its cost.
|
|
112
|
+
- The second label replaces the first; when the second run fails, the first label
|
|
113
|
+
stays. The label rows record the model that produced them, and a resumed run skips
|
|
114
|
+
prompts that were already escalated. The holdout check reports how many prompts were
|
|
115
|
+
escalated. `rescale --in A --out B` rewrites a label file with scaled labels (last
|
|
116
|
+
row per prompt).
|
|
91
117
|
- **Labeler check**: the labeler labels the 215 holdout prompts once, before any training
|
|
92
118
|
data is labeled. The report gives agreement with the owner's gold and the per-class recall.
|
|
93
119
|
- Labeling proceeds only when agreement is at least 80%.
|
|
@@ -114,6 +140,25 @@ prompts written by an agent, not by the owner. VS Code terminal notifications
|
|
|
114
140
|
weights and a weight decay of 1e-4.
|
|
115
141
|
- The output is the `.bin` weights, the `.json` sidecar and validation scores for
|
|
116
142
|
calibration.
|
|
143
|
+
- **Intent engine**: the same trainer with `--task intent` trains the intent stage
|
|
144
|
+
([system-one-intent.md](system-one-intent.md)).
|
|
145
|
+
- Labels come from `labels-intent.jsonl`, and owner corrections from
|
|
146
|
+
`owner-overrides-intent.jsonl`. `owner-excluded.jsonl` applies to both stages. The
|
|
147
|
+
catalog comes from the intent holdout.
|
|
148
|
+
- Targets are soft. The primary gets 1.0 when there is no secondary intent. Otherwise
|
|
149
|
+
the primary gets 0.6 and the secondary intents share 0.4 equally. `null` maps to
|
|
150
|
+
`unclassified` with 1.0.
|
|
151
|
+
- Validation scores keep each row's secondary intents for calibration.
|
|
152
|
+
- `--target scores` trains on the scored labels in `labels-intent-scores.jsonl`
|
|
153
|
+
instead. Each intent is its own binary target with its score as the soft value, and
|
|
154
|
+
the loss is a per-intent binary cross-entropy. `unclassified` scores 0.6 when the
|
|
155
|
+
primary is `null` and 0 otherwise. An owner override without scores counts as 0.6
|
|
156
|
+
for its primary and 0.4 for its secondary intents. The class weights come from the
|
|
157
|
+
score mass of each intent.
|
|
158
|
+
- The artifact format is unchanged. The provider still returns the softmax over the
|
|
159
|
+
same logits, which ranks the intents in relevance order, so the contract's
|
|
160
|
+
probability vector and the calibration below stay as they are. Handing the absolute
|
|
161
|
+
relevance bands to the skills stage is a separate contract change.
|
|
117
162
|
- The `tinyx` path below is kept for comparison. Its first checkpoint learned little beyond
|
|
118
163
|
class priors.
|
|
119
164
|
- **Model** (`tinyx`): a new `cua_s1` `tinyx` scorer built with `make_system`.
|
|
@@ -128,7 +173,12 @@ prompts written by an agent, not by the owner. VS Code terminal notifications
|
|
|
128
173
|
- The epoch with the lowest validation negative log-likelihood is kept.
|
|
129
174
|
- The checkpoint metadata records the dataset hashes, the config and the validation metrics.
|
|
130
175
|
- **Calibration**: `minConfidence` and `minMargin` are chosen on validation to maximize
|
|
131
|
-
coverage, subject to an accepted precision of at least 85% (the owner's advisory target).
|
|
176
|
+
coverage, subject to an accepted precision of at least 85% (the owner's advisory target).
|
|
177
|
+
For the intent stage (`--task intent`), accepted precision is the mean graded agreement
|
|
178
|
+
of accepted rows. A predicted secondary is any other intent whose probability reaches
|
|
179
|
+
`secondaryThreshold`. That threshold is chosen from 0.05 to 0.50 in steps of 0.05, to
|
|
180
|
+
maximize micro-F1 against the labeled secondary intents, and it is chosen before the
|
|
181
|
+
acceptance thresholds. The router currently uses the
|
|
132
182
|
Phase 0 defaults (0.9 and 0.2). Carrying calibrated thresholds into the manifest is a
|
|
133
183
|
separate contract change.
|
|
134
184
|
|
|
@@ -76,6 +76,13 @@ flowchart TD
|
|
|
76
76
|
Check -->|Pass| Handoff[Compact evidence + handoff]
|
|
77
77
|
```
|
|
78
78
|
|
|
79
|
+
Minor-concern batching is **not** a stage transition. Workers keep producing and
|
|
80
|
+
verifying the assigned artifact and include accumulated minor concerns in the next
|
|
81
|
+
natural report/handoff; the number of minor concerns alone never stops or returns a
|
|
82
|
+
worker. A material confirmed blocker may stop the current stage immediately. Repeated
|
|
83
|
+
matching execution failures use the separate Rule-of-3 path: zoom out, reflect, then
|
|
84
|
+
`RESUME` on a new approach or `ESCALATE` when a real decision is required.
|
|
85
|
+
|
|
79
86
|
At most two full replans are allowed before unresolved blockers are escalated rather than hidden behind endless restructuring.
|
|
80
87
|
|
|
81
88
|
## 5. Completion Boundary
|
package/fable-mode/SKILL.md
CHANGED
|
@@ -152,10 +152,10 @@ hasn't been confirmed present. Absence of evidence is not the finding; web
|
|
|
152
152
|
silence is never grounds for a warning against the user's firsthand information.
|
|
153
153
|
Confirm, then flag.
|
|
154
154
|
|
|
155
|
-
**Warning
|
|
156
|
-
|
|
157
|
-
|
|
158
|
-
|
|
155
|
+
**Warning batching.** Collect minor concerns and list them together in the next
|
|
156
|
+
natural report or handoff; their count alone MUST NOT stop, pause, or return the
|
|
157
|
+
worker/stage. An independently material, confirmed concern may stop the current
|
|
158
|
+
stage and be surfaced immediately.
|
|
159
159
|
|
|
160
160
|
**Find-and-replace safety.** Anchor substring replaces on word boundaries
|
|
161
161
|
(`\bword\b`, never bare `word` — a bare `edge` replace mangles `Ledger`). Prefer
|
|
@@ -41,10 +41,10 @@ Grep, diff, run it, or check the source directly. Never report a problem that
|
|
|
41
41
|
hasn't been confirmed present. Absence of evidence is not the finding. Confirm,
|
|
42
42
|
then flag.
|
|
43
43
|
|
|
44
|
-
**Warning
|
|
45
|
-
|
|
46
|
-
|
|
47
|
-
|
|
44
|
+
**Warning batching.** Collect minor concerns and list them together in the next
|
|
45
|
+
natural report or handoff; their count alone MUST NOT stop, pause, or return the
|
|
46
|
+
worker/stage. An independently material, confirmed concern may stop the current
|
|
47
|
+
stage and be surfaced immediately.
|
|
48
48
|
|
|
49
49
|
**Find-and-replace safety.** Anchor substring replaces on word boundaries
|
|
50
50
|
(`\bword\b`, never bare `word` — a bare `edge` replace mangles `Ledger`). Prefer
|
|
@@ -42,10 +42,10 @@ hasn't been confirmed present. Absence of evidence is not the finding; web
|
|
|
42
42
|
silence is never grounds for a warning against the user's firsthand information.
|
|
43
43
|
Confirm, then flag.
|
|
44
44
|
|
|
45
|
-
**Warning
|
|
46
|
-
|
|
47
|
-
|
|
48
|
-
|
|
45
|
+
**Warning batching.** Collect minor concerns and list them together in the next
|
|
46
|
+
natural report or handoff; their count alone MUST NOT stop, pause, or return the
|
|
47
|
+
worker/stage. An independently material, confirmed concern may stop the current
|
|
48
|
+
stage and be surfaced immediately.
|
|
49
49
|
|
|
50
50
|
**Find-and-replace safety.** Anchor substring replaces on word boundaries
|
|
51
51
|
(`\bword\b`, never bare `word` — a bare `edge` replace mangles `Ledger`). Prefer
|
|
@@ -7,7 +7,7 @@ description: >-
|
|
|
7
7
|
license: Apache-2.0
|
|
8
8
|
metadata:
|
|
9
9
|
author: Miya Daniel
|
|
10
|
-
version: 0.
|
|
10
|
+
version: 0.30.1
|
|
11
11
|
---
|
|
12
12
|
|
|
13
13
|
# Execution Guardrails
|
|
@@ -17,7 +17,7 @@ metadata:
|
|
|
17
17
|
| Component | Spec |
|
|
18
18
|
| :--- | :--- |
|
|
19
19
|
| **Trigger / Input** | Turn raising a warning or performing search-and-replace edits. |
|
|
20
|
-
| **Expected Output** | Verified warnings; batched caveats
|
|
20
|
+
| **Expected Output** | Verified warnings; batched caveats without count-based stops; corruption-checked edits. |
|
|
21
21
|
| **State Mutations** | None — governs execution quality for file-editing turns. |
|
|
22
22
|
| **Enforcement Gate** | Verify before flagging; validate edits post-write. |
|
|
23
23
|
|
|
@@ -35,9 +35,11 @@ metadata:
|
|
|
35
35
|
## Core Rules
|
|
36
36
|
|
|
37
37
|
1. **Verify before flag** — Confirm a problem exists before reporting it. Never convert absence of evidence into a warning.
|
|
38
|
-
2. **Warning
|
|
38
|
+
2. **Warning batching.** Collect minor concerns and list them together in the next natural report or handoff; their count alone MUST NOT stop, pause, or return the worker/stage. An independently material, confirmed concern may stop the current stage and be surfaced immediately.
|
|
39
39
|
3. **Find-and-replace safety** — Prefer structured edit tools over shell `sed`; anchor with unique context or `\bword\b` boundaries; verify file integrity post-edit. Never replace-all blindly.
|
|
40
40
|
|
|
41
|
+
Repeated matching execution failures remain governed by Rule-of-3 / `zoom-out`; warning batching never substitutes for that recovery path.
|
|
42
|
+
|
|
41
43
|
These rules are always-on, even when `fable-mode`'s staged loop is not running.
|
|
42
44
|
|
|
43
45
|
Deep dive: <this-skill-dir>/references/guardrail-rules.md
|
|
@@ -26,17 +26,21 @@ attempted and where it failed — not a vague appeal to difficulty.
|
|
|
26
26
|
|
|
27
27
|
## 2. Warning threshold
|
|
28
28
|
|
|
29
|
-
Across any run, minor concerns accumulate that
|
|
30
|
-
Keep a running count. At the threshold — **default three, tunable if
|
|
31
|
-
different number** —
|
|
29
|
+
Across any run, minor concerns accumulate that are not worth interrupting execution
|
|
30
|
+
for individually. Keep a running count. At the threshold — **default three, tunable if
|
|
31
|
+
the user sets a different number** — batch them for the next natural report or handoff.
|
|
32
|
+
The threshold changes presentation only: it MUST NOT stop, pause, return, or otherwise
|
|
33
|
+
end a worker/stage by count alone.
|
|
32
34
|
|
|
33
|
-
Rationale: three small things pointing the same direction
|
|
34
|
-
|
|
35
|
-
|
|
36
|
-
|
|
35
|
+
Rationale: three small things pointing the same direction are easier to evaluate as one
|
|
36
|
+
batch. Below threshold, keep working; a drip of trivial caveats is noise. At threshold,
|
|
37
|
+
keep working and preserve the batch for the report instead of creating a second circuit
|
|
38
|
+
breaker.
|
|
37
39
|
|
|
38
40
|
A concern that independently meets the verify-before-flag bar and is material on its own
|
|
39
|
-
|
|
41
|
+
may stop the current stage and be surfaced immediately. Repeated matching execution
|
|
42
|
+
failures are governed separately by Rule-of-3 -> zoom-out -> RESUME/ESCALATE. The
|
|
43
|
+
warning threshold governs minor-concern reporting only.
|
|
40
44
|
|
|
41
45
|
## 3. Find-and-replace safety
|
|
42
46
|
|