@miller-tech/uap 2.1.7 → 2.11.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/config/capacity-policy.json +32 -0
- package/config/classify-thresholds.json +9 -0
- package/config/supervise-policy.json +13 -0
- package/dist/.tsbuildinfo +1 -1
- package/dist/bin/cli.js +87 -2
- package/dist/bin/cli.js.map +1 -1
- package/dist/capacity/policy.d.ts +67 -0
- package/dist/capacity/policy.d.ts.map +1 -0
- package/dist/capacity/policy.js +133 -0
- package/dist/capacity/policy.js.map +1 -0
- package/dist/capacity/probe.d.ts +23 -0
- package/dist/capacity/probe.d.ts.map +1 -0
- package/dist/capacity/probe.js +93 -0
- package/dist/capacity/probe.js.map +1 -0
- package/dist/classify/baseline.d.ts +24 -0
- package/dist/classify/baseline.d.ts.map +1 -0
- package/dist/classify/baseline.js +140 -0
- package/dist/classify/baseline.js.map +1 -0
- package/dist/classify/index.d.ts +30 -0
- package/dist/classify/index.d.ts.map +1 -0
- package/dist/classify/index.js +84 -0
- package/dist/classify/index.js.map +1 -0
- package/dist/classify/shadow.d.ts +34 -0
- package/dist/classify/shadow.d.ts.map +1 -0
- package/dist/classify/shadow.js +33 -0
- package/dist/classify/shadow.js.map +1 -0
- package/dist/classify/types.d.ts +64 -0
- package/dist/classify/types.d.ts.map +1 -0
- package/dist/classify/types.js +37 -0
- package/dist/classify/types.js.map +1 -0
- package/dist/cli/classify.d.ts +24 -0
- package/dist/cli/classify.d.ts.map +1 -0
- package/dist/cli/classify.js +94 -0
- package/dist/cli/classify.js.map +1 -0
- package/dist/cli/deliver.d.ts.map +1 -1
- package/dist/cli/deliver.js +26 -0
- package/dist/cli/deliver.js.map +1 -1
- package/dist/cli/doctor.d.ts +9 -0
- package/dist/cli/doctor.d.ts.map +1 -0
- package/dist/cli/doctor.js +49 -0
- package/dist/cli/doctor.js.map +1 -0
- package/dist/cli/review.d.ts +12 -0
- package/dist/cli/review.d.ts.map +1 -0
- package/dist/cli/review.js +118 -0
- package/dist/cli/review.js.map +1 -0
- package/dist/cli/supervise.d.ts +8 -0
- package/dist/cli/supervise.d.ts.map +1 -0
- package/dist/cli/supervise.js +76 -0
- package/dist/cli/supervise.js.map +1 -0
- package/dist/delivery/convergence-loop.d.ts +8 -0
- package/dist/delivery/convergence-loop.d.ts.map +1 -1
- package/dist/delivery/convergence-loop.js +1 -0
- package/dist/delivery/convergence-loop.js.map +1 -1
- package/dist/delivery/delivery-result.d.ts.map +1 -1
- package/dist/delivery/delivery-result.js +5 -0
- package/dist/delivery/delivery-result.js.map +1 -1
- package/dist/delivery/gate-evidence.d.ts +143 -0
- package/dist/delivery/gate-evidence.d.ts.map +1 -0
- package/dist/delivery/gate-evidence.js +239 -0
- package/dist/delivery/gate-evidence.js.map +1 -0
- package/dist/delivery/index.d.ts +1 -0
- package/dist/delivery/index.d.ts.map +1 -1
- package/dist/delivery/index.js +1 -0
- package/dist/delivery/index.js.map +1 -1
- package/dist/evals/routing-corpus.d.ts +35 -0
- package/dist/evals/routing-corpus.d.ts.map +1 -0
- package/dist/evals/routing-corpus.js +142 -0
- package/dist/evals/routing-corpus.js.map +1 -0
- package/dist/evals/routing-eval.d.ts +104 -0
- package/dist/evals/routing-eval.d.ts.map +1 -0
- package/dist/evals/routing-eval.js +231 -0
- package/dist/evals/routing-eval.js.map +1 -0
- package/dist/evals/tfidf.d.ts +35 -0
- package/dist/evals/tfidf.d.ts.map +1 -0
- package/dist/evals/tfidf.js +132 -0
- package/dist/evals/tfidf.js.map +1 -0
- package/dist/review/prepass.d.ts +88 -0
- package/dist/review/prepass.d.ts.map +1 -0
- package/dist/review/prepass.js +428 -0
- package/dist/review/prepass.js.map +1 -0
- package/dist/review/visual-captures.d.ts +60 -0
- package/dist/review/visual-captures.d.ts.map +1 -0
- package/dist/review/visual-captures.js +222 -0
- package/dist/review/visual-captures.js.map +1 -0
- package/dist/supervise/assess.d.ts +73 -0
- package/dist/supervise/assess.d.ts.map +1 -0
- package/dist/supervise/assess.js +215 -0
- package/dist/supervise/assess.js.map +1 -0
- package/dist/supervise/config.d.ts +13 -0
- package/dist/supervise/config.d.ts.map +1 -0
- package/dist/supervise/config.js +81 -0
- package/dist/supervise/config.js.map +1 -0
- package/dist/supervise/loop.d.ts +78 -0
- package/dist/supervise/loop.d.ts.map +1 -0
- package/dist/supervise/loop.js +311 -0
- package/dist/supervise/loop.js.map +1 -0
- package/dist/supervise/observe.d.ts +31 -0
- package/dist/supervise/observe.d.ts.map +1 -0
- package/dist/supervise/observe.js +172 -0
- package/dist/supervise/observe.js.map +1 -0
- package/dist/supervise/policy.d.ts +16 -0
- package/dist/supervise/policy.d.ts.map +1 -0
- package/dist/supervise/policy.js +147 -0
- package/dist/supervise/policy.js.map +1 -0
- package/dist/supervise/types.d.ts +109 -0
- package/dist/supervise/types.d.ts.map +1 -0
- package/dist/supervise/types.js +11 -0
- package/dist/supervise/types.js.map +1 -0
- package/docs/INDEX.md +9 -3
- package/docs/architecture/adr/0001-automode-risk-floor.md +75 -0
- package/docs/architecture/adr/0002-evidence-bound-ship.md +81 -0
- package/docs/guides/CAPACITY_POLICY.md +98 -0
- package/docs/performance/automode-risk-calibration.md +72 -0
- package/docs/performance/reliability-ladder.md +78 -0
- package/docs/plans/system1-uplift-2026-09-18.md +191 -0
- package/docs/reference/automode-risk.md +134 -0
- package/docs/reference/evidence-bound-ship.md +145 -0
- package/docs/reference/review-prepass.md +77 -0
- package/docs/reference/routing-evals.md +71 -0
- package/docs/reference/semantic-supervisor.md +140 -0
- package/docs/reference/system1-classifier.md +103 -0
- package/docs/reference/visual-captures.md +113 -0
- package/package.json +5 -3
- package/src/policies/enforcers/__pycache__/_common.cpython-312.pyc +0 -0
- package/src/policies/enforcers/__pycache__/expert_review_required.cpython-312.pyc +0 -0
- package/src/policies/enforcers/__pycache__/quality_metrics_gate.cpython-312.pyc +0 -0
- package/src/policies/enforcers/enforcement_self_protect.py +25 -2
- package/src/policies/enforcers/evidence_bound_ship.py +257 -0
- package/src/policies/enforcers/expert_review_required.py +120 -0
- package/src/policies/schemas/policies/evidence-bound-ship.md +106 -0
- package/src/policies/schemas/policies/visual-verification.md +8 -0
- package/templates/hooks/__pycache__/deliver_autoroute.cpython-312.pyc +0 -0
- package/tools/agents/scripts/__pycache__/toolcall_path_normalizer.cpython-312.pyc +0 -0
- package/tools/agents/scripts/anthropic_proxy.py +782 -19
- package/tools/agents/scripts/project_telemetry.py +52 -0
- package/tools/agents/scripts/tool_risk.py +789 -0
- package/tools/agents/scripts/tool_risk_calibrate.py +255 -0
- package/tools/agents/tests/fixtures/tool_risk_cases.json +124 -0
- package/tools/agents/tests/test_enforcer_escape_hatches.py +13 -0
- package/tools/agents/tests/test_gate_evidence.py +56 -0
- package/tools/agents/tests/test_tool_risk.py +521 -0
- package/tools/agents/tests/test_verbatim_compact.py +629 -0
|
@@ -0,0 +1,147 @@
|
|
|
1
|
+
/**
|
|
2
|
+
* The deterministic supervisor policy. PURE — same inputs, same decision.
|
|
3
|
+
*
|
|
4
|
+
* Priority is safety-first and FIXED (do not reorder without review):
|
|
5
|
+
* 0. terminal status → FINISH when delivered, else CONTINUE (a run that
|
|
6
|
+
* is no longer running is never STOPped/ESCALATEd)
|
|
7
|
+
* 1. iteration bounds → STOP — outranks even human-need: a run past its
|
|
8
|
+
* budget stops regardless; a human-needed signal riding along is carried
|
|
9
|
+
* in evidence.alsoEscalate instead of becoming the action
|
|
10
|
+
* 2. human-needed → ESCALATE (a blocked mission burns budget silently)
|
|
11
|
+
* 3. off-track/stuck → RETRY while retries remain, else ESCALATE
|
|
12
|
+
* 4. completion-signaled → FINISH, ONLY on authoritative state
|
|
13
|
+
* (status === 'delivered'); the log-tail success regex alone never
|
|
14
|
+
* detaches supervision — a stuck mission whose model prints "mission
|
|
15
|
+
* complete" keeps being watched
|
|
16
|
+
* 5. verification-pending → VERIFY, NEVER twice consecutively (oscillation
|
|
17
|
+
* guard), and bounded by maxVerify: an unanswered verification falls
|
|
18
|
+
* into the stuck path (RETRY/ESCALATE)
|
|
19
|
+
* 6. otherwise → CONTINUE
|
|
20
|
+
*
|
|
21
|
+
* Every decision carries its reason and the evidence that fired it — the
|
|
22
|
+
* event ledger must be able to explain every action without re-running.
|
|
23
|
+
*/
|
|
24
|
+
import { runBudgetMinutes } from '../delivery/run-state.js';
|
|
25
|
+
function dim(dims, name) {
|
|
26
|
+
return dims.find((d) => d.name === name);
|
|
27
|
+
}
|
|
28
|
+
function fired(dims, name) {
|
|
29
|
+
return dim(dims, name)?.value === true;
|
|
30
|
+
}
|
|
31
|
+
function scoreOf(dims, name) {
|
|
32
|
+
const v = dim(dims, name)?.value;
|
|
33
|
+
return typeof v === 'number' ? v : undefined;
|
|
34
|
+
}
|
|
35
|
+
/**
|
|
36
|
+
* The budget the STOP branch enforces: the LARGER of the supervisor policy
|
|
37
|
+
* and the deliver run's own budget (`UAP_DELIVER_MAX_MINUTES` /
|
|
38
|
+
* `.uap.json delivery.maxRunMinutes` / the calibrated default). An
|
|
39
|
+
* operator-raised deliver budget must not be overruled by the supervisor's
|
|
40
|
+
* static policy number; a LOWER operator budget makes the deliver loop stop
|
|
41
|
+
* itself first anyway, so taking the max never extends a mission past what
|
|
42
|
+
* its owner granted. Impure (reads env/config) — callers pass the result in
|
|
43
|
+
* via an adjusted cfg so decide() itself stays pure.
|
|
44
|
+
*/
|
|
45
|
+
export function effectiveMaxMinutes(cfg, projectRoot) {
|
|
46
|
+
return Math.max(cfg.maxMinutes, runBudgetMinutes(projectRoot));
|
|
47
|
+
}
|
|
48
|
+
/** RETRY while the budget remains, ESCALATE when it is spent. */
|
|
49
|
+
function retryOrEscalate(reasonKind, state, cfg, evidence) {
|
|
50
|
+
const withBudget = { ...evidence, retries: state.retries, maxRetries: cfg.maxRetries };
|
|
51
|
+
if (state.retries < cfg.maxRetries) {
|
|
52
|
+
return { action: 'RETRY', reason: `${reasonKind} — retry budget remains`, evidence: withBudget };
|
|
53
|
+
}
|
|
54
|
+
return { action: 'ESCALATE', reason: `${reasonKind} — retries exhausted`, evidence: withBudget };
|
|
55
|
+
}
|
|
56
|
+
export function decide(obs, dims, state, cfg) {
|
|
57
|
+
// (0) Terminal runs: nothing left to supervise. NEVER STOP/ESCALATE them —
|
|
58
|
+
// a delivered run's log may still contain scary text from earlier turns.
|
|
59
|
+
if (obs.status !== 'running') {
|
|
60
|
+
if (obs.status === 'delivered') {
|
|
61
|
+
return {
|
|
62
|
+
action: 'FINISH',
|
|
63
|
+
reason: 'run delivered',
|
|
64
|
+
evidence: { status: obs.status, turnsCompleted: obs.turnsCompleted },
|
|
65
|
+
};
|
|
66
|
+
}
|
|
67
|
+
return {
|
|
68
|
+
action: 'CONTINUE',
|
|
69
|
+
reason: `run is ${obs.status} — nothing to supervise`,
|
|
70
|
+
evidence: { status: obs.status },
|
|
71
|
+
};
|
|
72
|
+
}
|
|
73
|
+
const humanNeeded = fired(dims, 'human-needed');
|
|
74
|
+
// (1) Iteration bounds. STOP outranks ESCALATE: an over-budget mission that
|
|
75
|
+
// is ALSO waiting on a human still stops — the human-need is recorded as an
|
|
76
|
+
// alsoEscalate note, not acted on as the decision.
|
|
77
|
+
if (obs.turnsCompleted > cfg.maxTurns || obs.elapsedMinutes > cfg.maxMinutes) {
|
|
78
|
+
return {
|
|
79
|
+
action: 'STOP',
|
|
80
|
+
reason: 'iteration bounds exceeded',
|
|
81
|
+
evidence: {
|
|
82
|
+
turnsCompleted: obs.turnsCompleted,
|
|
83
|
+
maxTurns: cfg.maxTurns,
|
|
84
|
+
elapsedMinutes: Math.round(obs.elapsedMinutes * 10) / 10,
|
|
85
|
+
maxMinutes: cfg.maxMinutes,
|
|
86
|
+
...(humanNeeded ? { alsoEscalate: 'human input required' } : {}),
|
|
87
|
+
},
|
|
88
|
+
};
|
|
89
|
+
}
|
|
90
|
+
// (2) Human need: a blocked mission burns its whole budget silently.
|
|
91
|
+
if (humanNeeded) {
|
|
92
|
+
return {
|
|
93
|
+
action: 'ESCALATE',
|
|
94
|
+
reason: 'human input required',
|
|
95
|
+
evidence: { humanNeeded: true, tail: obs.recentLogTail.slice(-200) },
|
|
96
|
+
};
|
|
97
|
+
}
|
|
98
|
+
// (3) Off-track or stuck: retry while the budget remains, then escalate.
|
|
99
|
+
const offTrack = fired(dims, 'off-track');
|
|
100
|
+
const stuck = fired(dims, 'stuck-loop');
|
|
101
|
+
if (offTrack || stuck) {
|
|
102
|
+
return retryOrEscalate('off-track/stuck', state, cfg, {
|
|
103
|
+
offTrack,
|
|
104
|
+
stuckLoop: stuck,
|
|
105
|
+
progressStalled: fired(dims, 'progress-stalled'),
|
|
106
|
+
failures: obs.failures,
|
|
107
|
+
});
|
|
108
|
+
}
|
|
109
|
+
// (4) Completion: FINISH happens ONLY on authoritative state — branch (0)
|
|
110
|
+
// returns FINISH for status 'delivered'. A completion-signaled dimension
|
|
111
|
+
// firing from the log tail alone (a stuck model printing "mission
|
|
112
|
+
// complete") is deliberately ignored here: supervision stays attached.
|
|
113
|
+
// (5) Verification, oscillation-guarded AND budgeted. VERIFY never fires
|
|
114
|
+
// twice consecutively; once the verify budget is spent without test
|
|
115
|
+
// evidence appearing, the run is treated as stuck.
|
|
116
|
+
if (fired(dims, 'verification-pending')) {
|
|
117
|
+
if (state.lastAction !== 'VERIFY') {
|
|
118
|
+
return {
|
|
119
|
+
action: 'VERIFY',
|
|
120
|
+
reason: 'checkpoint exists without test evidence in the log tail',
|
|
121
|
+
evidence: { hasCheckpoint: obs.hasCheckpoint, lastAction: state.lastAction, verifyCount: state.verifyCount },
|
|
122
|
+
};
|
|
123
|
+
}
|
|
124
|
+
if (state.verifyCount >= cfg.maxVerify) {
|
|
125
|
+
return retryOrEscalate('verification unanswered', state, cfg, {
|
|
126
|
+
verifyCount: state.verifyCount,
|
|
127
|
+
maxVerify: cfg.maxVerify,
|
|
128
|
+
});
|
|
129
|
+
}
|
|
130
|
+
return {
|
|
131
|
+
action: 'CONTINUE',
|
|
132
|
+
reason: 'verify already requested, awaiting result',
|
|
133
|
+
evidence: { verifyCount: state.verifyCount, maxVerify: cfg.maxVerify },
|
|
134
|
+
};
|
|
135
|
+
}
|
|
136
|
+
// (6) Healthy.
|
|
137
|
+
return {
|
|
138
|
+
action: 'CONTINUE',
|
|
139
|
+
reason: 'within bounds',
|
|
140
|
+
evidence: {
|
|
141
|
+
errorDensity: scoreOf(dims, 'error-density'),
|
|
142
|
+
iterationPressure: scoreOf(dims, 'iteration-pressure'),
|
|
143
|
+
risk: scoreOf(dims, 'risk'),
|
|
144
|
+
},
|
|
145
|
+
};
|
|
146
|
+
}
|
|
147
|
+
//# sourceMappingURL=policy.js.map
|
|
@@ -0,0 +1 @@
|
|
|
1
|
+
{"version":3,"file":"policy.js","sourceRoot":"","sources":["../../src/supervise/policy.ts"],"names":[],"mappings":"AAAA;;;;;;;;;;;;;;;;;;;;;;GAsBG;AACH,OAAO,EAAE,gBAAgB,EAAE,MAAM,0BAA0B,CAAC;AAG5D,SAAS,GAAG,CAAC,IAA2B,EAAE,IAAY;IACpD,OAAO,IAAI,CAAC,IAAI,CAAC,CAAC,CAAC,EAAE,EAAE,CAAC,CAAC,CAAC,IAAI,KAAK,IAAI,CAAC,CAAC;AAC3C,CAAC;AAED,SAAS,KAAK,CAAC,IAA2B,EAAE,IAAY;IACtD,OAAO,GAAG,CAAC,IAAI,EAAE,IAAI,CAAC,EAAE,KAAK,KAAK,IAAI,CAAC;AACzC,CAAC;AAED,SAAS,OAAO,CAAC,IAA2B,EAAE,IAAY;IACxD,MAAM,CAAC,GAAG,GAAG,CAAC,IAAI,EAAE,IAAI,CAAC,EAAE,KAAK,CAAC;IACjC,OAAO,OAAO,CAAC,KAAK,QAAQ,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,SAAS,CAAC;AAC/C,CAAC;AAKD;;;;;;;;;GASG;AACH,MAAM,UAAU,mBAAmB,CAAC,GAAqB,EAAE,WAAmB;IAC5E,OAAO,IAAI,CAAC,GAAG,CAAC,GAAG,CAAC,UAAU,EAAE,gBAAgB,CAAC,WAAW,CAAC,CAAC,CAAC;AACjE,CAAC;AAED,iEAAiE;AACjE,SAAS,eAAe,CACtB,UAAkB,EAClB,KAAkB,EAClB,GAAqB,EACrB,QAAiC;IAEjC,MAAM,UAAU,GAAG,EAAE,GAAG,QAAQ,EAAE,OAAO,EAAE,KAAK,CAAC,OAAO,EAAE,UAAU,EAAE,GAAG,CAAC,UAAU,EAAE,CAAC;IACvF,IAAI,KAAK,CAAC,OAAO,GAAG,GAAG,CAAC,UAAU,EAAE,CAAC;QACnC,OAAO,EAAE,MAAM,EAAE,OAAO,EAAE,MAAM,EAAE,GAAG,UAAU,yBAAyB,EAAE,QAAQ,EAAE,UAAU,EAAE,CAAC;IACnG,CAAC;IACD,OAAO,EAAE,MAAM,EAAE,UAAU,EAAE,MAAM,EAAE,GAAG,UAAU,sBAAsB,EAAE,QAAQ,EAAE,UAAU,EAAE,CAAC;AACnG,CAAC;AAED,MAAM,UAAU,MAAM,CACpB,GAAgB,EAChB,IAA2B,EAC3B,KAAkB,EAClB,GAAqB;IAErB,2EAA2E;IAC3E,yEAAyE;IACzE,IAAI,GAAG,CAAC,MAAM,KAAK,SAAS,EAAE,CAAC;QAC7B,IAAI,GAAG,CAAC,MAAM,KAAK,WAAW,EAAE,CAAC;YAC/B,OAAO;gBACL,MAAM,EAAE,QAAQ;gBAChB,MAAM,EAAE,eAAe;gBACvB,QAAQ,EAAE,EAAE,MAAM,EAAE,GAAG,CAAC,MAAM,EAAE,cAAc,EAAE,GAAG,CAAC,cAAc,EAAE;aACrE,CAAC;QACJ,CAAC;QACD,OAAO;YACL,MAAM,EAAE,UAAU;YAClB,MAAM,EAAE,UAAU,GAAG,CAAC,MAAM,yBAAyB;YACrD,QAAQ,EAAE,EAAE,MAAM,EAAE,GAAG,CAAC,MAAM,EAAE;SACjC,CAAC;IACJ,CAAC;IAED,MAAM,WAAW,GAAG,KAAK,CAAC,IAAI,EAAE,cAAc,CAAC,CAAC;IAEhD,4EAA4E;IAC5E,4EAA4E;IAC5E,mDAAmD;IACnD,IAAI,GAAG,CAAC,cAAc,GAAG,GAAG,CAAC,QAAQ,IAAI,GAAG,CAAC,cAAc,GAAG,GAAG,CAAC,UAAU,EAAE,CAAC;QAC7E,OAAO;YACL,MAAM,EAAE,MAAM;YACd,MAAM,EAAE,2BAA2B;YACnC,QAAQ,EAAE;gBACR,cAAc,EAAE,GAAG,CAAC,cAAc;gBAClC,QAAQ,EAAE,GAAG,CAAC,QAAQ;gBACtB,cAAc,EAAE,IAAI,CAAC,KAAK,CAAC,GAAG,CAAC,cAAc,GAAG,EAAE,CAAC,GAAG,EAAE;gBACxD,UAAU,EAAE,GAAG,CAAC,UAAU;gBAC1B,GAAG,CAAC,WAAW,CAAC,CAAC,CAAC,EAAE,YAAY,EAAE,sBAAsB,EAAE,CAAC,CAAC,CAAC,EAAE,CAAC;aACjE;SACF,CAAC;IACJ,CAAC;IAED,qEAAqE;IACrE,IAAI,WAAW,EAAE,CAAC;QAChB,OAAO;YACL,MAAM,EAAE,UAAU;YAClB,MAAM,EAAE,sBAAsB;YAC9B,QAAQ,EAAE,EAAE,WAAW,EAAE,IAAI,EAAE,IAAI,EAAE,GAAG,CAAC,aAAa,CAAC,KAAK,CAAC,CAAC,GAAG,CAAC,EAAE;SACrE,CAAC;IACJ,CAAC;IAED,yEAAyE;IACzE,MAAM,QAAQ,GAAG,KAAK,CAAC,IAAI,EAAE,WAAW,CAAC,CAAC;IAC1C,MAAM,KAAK,GAAG,KAAK,CAAC,IAAI,EAAE,YAAY,CAAC,CAAC;IACxC,IAAI,QAAQ,IAAI,KAAK,EAAE,CAAC;QACtB,OAAO,eAAe,CAAC,iBAAiB,EAAE,KAAK,EAAE,GAAG,EAAE;YACpD,QAAQ;YACR,SAAS,EAAE,KAAK;YAChB,eAAe,EAAE,KAAK,CAAC,IAAI,EAAE,kBAAkB,CAAC;YAChD,QAAQ,EAAE,GAAG,CAAC,QAAQ;SACvB,CAAC,CAAC;IACL,CAAC;IAED,0EAA0E;IAC1E,yEAAyE;IACzE,kEAAkE;IAClE,uEAAuE;IAEvE,yEAAyE;IACzE,oEAAoE;IACpE,mDAAmD;IACnD,IAAI,KAAK,CAAC,IAAI,EAAE,sBAAsB,CAAC,EAAE,CAAC;QACxC,IAAI,KAAK,CAAC,UAAU,KAAK,QAAQ,EAAE,CAAC;YAClC,OAAO;gBACL,MAAM,EAAE,QAAQ;gBAChB,MAAM,EAAE,yDAAyD;gBACjE,QAAQ,EAAE,EAAE,aAAa,EAAE,GAAG,CAAC,aAAa,EAAE,UAAU,EAAE,KAAK,CAAC,UAAU,EAAE,WAAW,EAAE,KAAK,CAAC,WAAW,EAAE;aAC7G,CAAC;QACJ,CAAC;QACD,IAAI,KAAK,CAAC,WAAW,IAAI,GAAG,CAAC,SAAS,EAAE,CAAC;YACvC,OAAO,eAAe,CAAC,yBAAyB,EAAE,KAAK,EAAE,GAAG,EAAE;gBAC5D,WAAW,EAAE,KAAK,CAAC,WAAW;gBAC9B,SAAS,EAAE,GAAG,CAAC,SAAS;aACzB,CAAC,CAAC;QACL,CAAC;QACD,OAAO;YACL,MAAM,EAAE,UAAU;YAClB,MAAM,EAAE,2CAA2C;YACnD,QAAQ,EAAE,EAAE,WAAW,EAAE,KAAK,CAAC,WAAW,EAAE,SAAS,EAAE,GAAG,CAAC,SAAS,EAAE;SACvE,CAAC;IACJ,CAAC;IAED,eAAe;IACf,OAAO;QACL,MAAM,EAAE,UAAU;QAClB,MAAM,EAAE,eAAe;QACvB,QAAQ,EAAE;YACR,YAAY,EAAE,OAAO,CAAC,IAAI,EAAE,eAAe,CAAC;YAC5C,iBAAiB,EAAE,OAAO,CAAC,IAAI,EAAE,oBAAoB,CAAC;YACtD,IAAI,EAAE,OAAO,CAAC,IAAI,EAAE,MAAM,CAAC;SAC5B;KACF,CAAC;AACJ,CAAC"}
|
|
@@ -0,0 +1,109 @@
|
|
|
1
|
+
/**
|
|
2
|
+
* Semantic supervisor for `uap deliver` missions — shared types.
|
|
3
|
+
*
|
|
4
|
+
* The supervisor WATCHES a run; it does not drive it. Observations are bounded
|
|
5
|
+
* (log tails capped, diffs capped, git calls time-boxed), assessments are pure
|
|
6
|
+
* functions over those observations, and the policy is deterministic with
|
|
7
|
+
* reviewed thresholds (config/supervise-policy.json). Models may only ASSESS
|
|
8
|
+
* (via the injectable classifier); they never decide.
|
|
9
|
+
*/
|
|
10
|
+
export type SupervisorAction = 'CONTINUE' | 'STOP' | 'RETRY' | 'VERIFY' | 'FINISH' | 'ESCALATE';
|
|
11
|
+
export type SupervisedRunStatus = 'running' | 'delivered' | 'failed' | 'interrupted';
|
|
12
|
+
/**
|
|
13
|
+
* One bounded snapshot of a deliver run. Everything here is cheap to collect
|
|
14
|
+
* and safe to hand to a classifier: no unbounded file contents, no secrets.
|
|
15
|
+
*
|
|
16
|
+
* Beyond the minimal contract (runId/status/elapsed/turns/failures/tail/git)
|
|
17
|
+
* this carries three derived fields the PURE dimension functions need so
|
|
18
|
+
* observation stays the only impure step: `instruction` (off-track keyword
|
|
19
|
+
* matching), `minutesSinceUpdate` (stall detection), and `hasCheckpoint`
|
|
20
|
+
* (verification-pending).
|
|
21
|
+
*/
|
|
22
|
+
export interface Observation {
|
|
23
|
+
runId: string;
|
|
24
|
+
status: SupervisedRunStatus;
|
|
25
|
+
/** Mission instruction (already length-clamped by run-state on load). */
|
|
26
|
+
instruction: string;
|
|
27
|
+
/** Wall-clock minutes since the run was created. */
|
|
28
|
+
elapsedMinutes: number;
|
|
29
|
+
/** Minutes since the run state was last persisted (stall signal). */
|
|
30
|
+
minutesSinceUpdate: number;
|
|
31
|
+
/** Last completed loop turn (0 before the first checkpoint). */
|
|
32
|
+
turnsCompleted: number;
|
|
33
|
+
/** Turns recorded as not-passed in the checkpoint history. */
|
|
34
|
+
failures: number;
|
|
35
|
+
/** A loop checkpoint exists (the run has completed at least one turn). */
|
|
36
|
+
hasCheckpoint: boolean;
|
|
37
|
+
/** Tail of the newest matching deliver log, capped at 8KB. '' when absent. */
|
|
38
|
+
recentLogTail: string;
|
|
39
|
+
/** `git status --porcelain` line count; undefined when git is unavailable. */
|
|
40
|
+
gitDirtyFiles?: number;
|
|
41
|
+
/** `git diff --stat`, capped; undefined when git is unavailable. */
|
|
42
|
+
diffStat?: string;
|
|
43
|
+
}
|
|
44
|
+
export interface DimensionAssessment {
|
|
45
|
+
name: string;
|
|
46
|
+
/** boolean for flag dimensions, 1-5 for score dimensions. */
|
|
47
|
+
value: boolean | number;
|
|
48
|
+
/** Estimated probability the assessment is correct (0-1). */
|
|
49
|
+
probability: number;
|
|
50
|
+
/** Confidence in THIS answer (0-1); low confidence defers to the heuristic. */
|
|
51
|
+
confidence: number;
|
|
52
|
+
/** True when the assessor abstains (insufficient evidence). */
|
|
53
|
+
defer: boolean;
|
|
54
|
+
source: 'heuristic' | 'classifier';
|
|
55
|
+
}
|
|
56
|
+
export interface Decision {
|
|
57
|
+
action: SupervisorAction;
|
|
58
|
+
reason: string;
|
|
59
|
+
/** The dimension values and counters that fired, for the event ledger. */
|
|
60
|
+
evidence: Record<string, unknown>;
|
|
61
|
+
}
|
|
62
|
+
/**
|
|
63
|
+
* Reviewed policy thresholds (config/supervise-policy.json). Validated loudly
|
|
64
|
+
* at load — a missing or out-of-range policy REFUSES to run rather than
|
|
65
|
+
* silently supervising with invented numbers.
|
|
66
|
+
*/
|
|
67
|
+
export interface SupervisorConfig {
|
|
68
|
+
version: 1;
|
|
69
|
+
/** running + no state update for this long ⇒ progress-stalled. */
|
|
70
|
+
stallMinutes: number;
|
|
71
|
+
/** elapsed beyond this ⇒ STOP (mirrors the calibrated 120m deliver budget). */
|
|
72
|
+
maxMinutes: number;
|
|
73
|
+
/** turns beyond this ⇒ STOP. */
|
|
74
|
+
maxTurns: number;
|
|
75
|
+
/** RETRY budget before off-track/stuck escalates. */
|
|
76
|
+
maxRetries: number;
|
|
77
|
+
/** checkpoint failures at/above this ⇒ stuck-loop. */
|
|
78
|
+
maxFailures: number;
|
|
79
|
+
/** minimum gap between assessments (change-triggered). */
|
|
80
|
+
debounceMs: number;
|
|
81
|
+
/** maximum gap between assessments (periodic heartbeat). */
|
|
82
|
+
intervalMs: number;
|
|
83
|
+
/** classifier answers below this confidence are ignored (fail closed). */
|
|
84
|
+
classifierConfidenceMin: number;
|
|
85
|
+
/** escalation-risk answers below this probability are ignored. */
|
|
86
|
+
classifierTau: number;
|
|
87
|
+
/** VERIFY actions issued without test evidence before stuck-path takeover. */
|
|
88
|
+
maxVerify: number;
|
|
89
|
+
}
|
|
90
|
+
/** State persisted per supervised run at .uap/supervise/<runId>/state.json. */
|
|
91
|
+
export interface SupervisorState {
|
|
92
|
+
version: 1;
|
|
93
|
+
runId: string;
|
|
94
|
+
startedAt: string;
|
|
95
|
+
updatedAt: string;
|
|
96
|
+
/** Assessment cycles completed. */
|
|
97
|
+
assessments: number;
|
|
98
|
+
/** RETRY actions issued so far (drives the retry→escalate boundary). */
|
|
99
|
+
retries: number;
|
|
100
|
+
/** VERIFY actions issued since test evidence was last seen. */
|
|
101
|
+
verifyCount: number;
|
|
102
|
+
/** Consecutive suppressed ESCALATE repeats (bounded by the loop). */
|
|
103
|
+
suppressedEscalations: number;
|
|
104
|
+
/** Last action taken — the oscillation guard's memory. */
|
|
105
|
+
lastAction: SupervisorAction | null;
|
|
106
|
+
lastActionAt: string | null;
|
|
107
|
+
lastDecision?: Decision;
|
|
108
|
+
}
|
|
109
|
+
//# sourceMappingURL=types.d.ts.map
|
|
@@ -0,0 +1 @@
|
|
|
1
|
+
{"version":3,"file":"types.d.ts","sourceRoot":"","sources":["../../src/supervise/types.ts"],"names":[],"mappings":"AAAA;;;;;;;;GAQG;AAEH,MAAM,MAAM,gBAAgB,GAAG,UAAU,GAAG,MAAM,GAAG,OAAO,GAAG,QAAQ,GAAG,QAAQ,GAAG,UAAU,CAAC;AAEhG,MAAM,MAAM,mBAAmB,GAAG,SAAS,GAAG,WAAW,GAAG,QAAQ,GAAG,aAAa,CAAC;AAErF;;;;;;;;;GASG;AACH,MAAM,WAAW,WAAW;IAC1B,KAAK,EAAE,MAAM,CAAC;IACd,MAAM,EAAE,mBAAmB,CAAC;IAC5B,yEAAyE;IACzE,WAAW,EAAE,MAAM,CAAC;IACpB,oDAAoD;IACpD,cAAc,EAAE,MAAM,CAAC;IACvB,qEAAqE;IACrE,kBAAkB,EAAE,MAAM,CAAC;IAC3B,gEAAgE;IAChE,cAAc,EAAE,MAAM,CAAC;IACvB,8DAA8D;IAC9D,QAAQ,EAAE,MAAM,CAAC;IACjB,0EAA0E;IAC1E,aAAa,EAAE,OAAO,CAAC;IACvB,8EAA8E;IAC9E,aAAa,EAAE,MAAM,CAAC;IACtB,8EAA8E;IAC9E,aAAa,CAAC,EAAE,MAAM,CAAC;IACvB,oEAAoE;IACpE,QAAQ,CAAC,EAAE,MAAM,CAAC;CACnB;AAED,MAAM,WAAW,mBAAmB;IAClC,IAAI,EAAE,MAAM,CAAC;IACb,6DAA6D;IAC7D,KAAK,EAAE,OAAO,GAAG,MAAM,CAAC;IACxB,6DAA6D;IAC7D,WAAW,EAAE,MAAM,CAAC;IACpB,+EAA+E;IAC/E,UAAU,EAAE,MAAM,CAAC;IACnB,+DAA+D;IAC/D,KAAK,EAAE,OAAO,CAAC;IACf,MAAM,EAAE,WAAW,GAAG,YAAY,CAAC;CACpC;AAED,MAAM,WAAW,QAAQ;IACvB,MAAM,EAAE,gBAAgB,CAAC;IACzB,MAAM,EAAE,MAAM,CAAC;IACf,0EAA0E;IAC1E,QAAQ,EAAE,MAAM,CAAC,MAAM,EAAE,OAAO,CAAC,CAAC;CACnC;AAED;;;;GAIG;AACH,MAAM,WAAW,gBAAgB;IAC/B,OAAO,EAAE,CAAC,CAAC;IACX,kEAAkE;IAClE,YAAY,EAAE,MAAM,CAAC;IACrB,+EAA+E;IAC/E,UAAU,EAAE,MAAM,CAAC;IACnB,gCAAgC;IAChC,QAAQ,EAAE,MAAM,CAAC;IACjB,qDAAqD;IACrD,UAAU,EAAE,MAAM,CAAC;IACnB,sDAAsD;IACtD,WAAW,EAAE,MAAM,CAAC;IACpB,0DAA0D;IAC1D,UAAU,EAAE,MAAM,CAAC;IACnB,4DAA4D;IAC5D,UAAU,EAAE,MAAM,CAAC;IACnB,0EAA0E;IAC1E,uBAAuB,EAAE,MAAM,CAAC;IAChC,kEAAkE;IAClE,aAAa,EAAE,MAAM,CAAC;IACtB,8EAA8E;IAC9E,SAAS,EAAE,MAAM,CAAC;CACnB;AAED,+EAA+E;AAC/E,MAAM,WAAW,eAAe;IAC9B,OAAO,EAAE,CAAC,CAAC;IACX,KAAK,EAAE,MAAM,CAAC;IACd,SAAS,EAAE,MAAM,CAAC;IAClB,SAAS,EAAE,MAAM,CAAC;IAClB,mCAAmC;IACnC,WAAW,EAAE,MAAM,CAAC;IACpB,wEAAwE;IACxE,OAAO,EAAE,MAAM,CAAC;IAChB,+DAA+D;IAC/D,WAAW,EAAE,MAAM,CAAC;IACpB,qEAAqE;IACrE,qBAAqB,EAAE,MAAM,CAAC;IAC9B,0DAA0D;IAC1D,UAAU,EAAE,gBAAgB,GAAG,IAAI,CAAC;IACpC,YAAY,EAAE,MAAM,GAAG,IAAI,CAAC;IAC5B,YAAY,CAAC,EAAE,QAAQ,CAAC;CACzB"}
|
|
@@ -0,0 +1,11 @@
|
|
|
1
|
+
/**
|
|
2
|
+
* Semantic supervisor for `uap deliver` missions — shared types.
|
|
3
|
+
*
|
|
4
|
+
* The supervisor WATCHES a run; it does not drive it. Observations are bounded
|
|
5
|
+
* (log tails capped, diffs capped, git calls time-boxed), assessments are pure
|
|
6
|
+
* functions over those observations, and the policy is deterministic with
|
|
7
|
+
* reviewed thresholds (config/supervise-policy.json). Models may only ASSESS
|
|
8
|
+
* (via the injectable classifier); they never decide.
|
|
9
|
+
*/
|
|
10
|
+
export {};
|
|
11
|
+
//# sourceMappingURL=types.js.map
|
|
@@ -0,0 +1 @@
|
|
|
1
|
+
{"version":3,"file":"types.js","sourceRoot":"","sources":["../../src/supervise/types.ts"],"names":[],"mappings":"AAAA;;;;;;;;GAQG"}
|
package/docs/INDEX.md
CHANGED
|
@@ -18,12 +18,13 @@ UAP is organized like a delivery line. If you know which part of the pipeline yo
|
|
|
18
18
|
| Station | The break it prevents | Start with |
|
|
19
19
|
|---|---|---|
|
|
20
20
|
| **Intake** — understand the work | Amnesiac sessions, invented scope | [Memory](guides/MEMORY.md) · [Reactor](design/UAP_REACTOR.md) |
|
|
21
|
-
| **Prep / routing** — right job, right station | Wrong approach or wrong-sized model | [Multi-Model Routing](guides/MULTI_MODEL.md) · [Patterns](reference/PATTERNS.md) · [Droids & Skills](guides/DROIDS_AND_SKILLS.md) |
|
|
21
|
+
| **Prep / routing** — right job, right station | Wrong approach or wrong-sized model | [Multi-Model Routing](guides/MULTI_MODEL.md) · [Patterns](reference/PATTERNS.md) · [Droids & Skills](guides/DROIDS_AND_SKILLS.md) · [System-1 Classifier](reference/system1-classifier.md) (`uap classify`) |
|
|
22
22
|
| **Isolation** — a bench per job | Editing `main`, clobbering files | [Worktree Workflow](guides/WORKTREE_WORKFLOW.md) |
|
|
23
23
|
| **Build** — make the thing | Plausible-but-wrong code, stubs, empty output | [`uap deliver`](guides/DELIVER.md) · [Local Models](guides/LOCAL_MODELS.md) · [Inference Proxy](guides/PROXY.md) |
|
|
24
|
-
| **QC / verify** — prove it runs | "Done" on code that never ran | [`uap deliver`](guides/DELIVER.md) · [Policies](guides/POLICIES.md) · [Quality gate](../CLAUDE.md) (`uap quality check`) |
|
|
25
|
-
| **Coordination** — many workers, one floor | Parallel agents colliding, stale branches overwriting landed work | [Parallel Agents](guides/PARALLEL_AGENTS.md) · [Coordination](guides/COORDINATION.md) · [Deploy Batching](guides/DEPLOY_BATCHING.md) |
|
|
24
|
+
| **QC / verify** — prove it runs | "Done" on code that never ran | [`uap deliver`](guides/DELIVER.md) · [Policies](guides/POLICIES.md) · [Quality gate](../CLAUDE.md) (`uap quality check`) · [AutoMode risk](reference/automode-risk.md) · [Evidence-bound ship](reference/evidence-bound-ship.md) |
|
|
25
|
+
| **Coordination** — many workers, one floor | Parallel agents colliding, stale branches overwriting landed work | [Parallel Agents](guides/PARALLEL_AGENTS.md) · [Coordination](guides/COORDINATION.md) · [Deploy Batching](guides/DEPLOY_BATCHING.md) · [Semantic Supervisor](reference/semantic-supervisor.md) (`uap supervise`) |
|
|
26
26
|
| **Shipping** — out the door safely | Regressions, red CI, skipped bumps | [Worktree Workflow](guides/WORKTREE_WORKFLOW.md) · [Policies](guides/POLICIES.md) |
|
|
27
|
+
| **Capacity** — services inside their budgets | Silent OOMs, crash loops, headroom loss | [Capacity Policy](guides/CAPACITY_POLICY.md) (`uap doctor`) |
|
|
27
28
|
| **Feedback** — the floor learns | The same mistake every session | [Memory](guides/MEMORY.md) · [Self-Harness](design/SELF_HARNESS.md) · [Self-Tuning](guides/SELF_TUNING.md) |
|
|
28
29
|
| *Cross-cutting* — the whole line | Ignored rules, bloated context | [Policies](guides/POLICIES.md) · [MCP Router](guides/MCP_ROUTER.md) |
|
|
29
30
|
|
|
@@ -87,7 +88,10 @@ Full map: **[The UAP Delivery Pipeline](guides/DELIVERY_PIPELINE.md)**.
|
|
|
87
88
|
| [API](reference/API.md) | Programmatic API surface |
|
|
88
89
|
| [Features](reference/FEATURES.md) | Full feature catalog, mapped to the pipeline stages |
|
|
89
90
|
| [Patterns](reference/PATTERNS.md) | The 23 Terminal-Bench patterns |
|
|
91
|
+
| [Routing Evals](reference/routing-evals.md) | CI-enforced rank-1 routing evals over the pattern/droid/skill registries — planted traps, negative cases, 90% threshold |
|
|
90
92
|
| [Platforms](reference/PLATFORMS.md) | The 9 supported harnesses + support matrix |
|
|
93
|
+
| [Review Pre-Pass](reference/review-prepass.md) | Deterministic ruleset scanner (`uap review prepass`) that feeds line-anchored findings to the parallel review protocol |
|
|
94
|
+
| [Visual Captures](reference/visual-captures.md) | Before/after capture binding for UI diffs (`uap review captures`) — ship-time gate 7 enforcement |
|
|
91
95
|
| [Configuration](reference/CONFIGURATION.md) | All config files and env vars |
|
|
92
96
|
| [**Configuration Reference**](reference/CONFIGURATION_REFERENCE.md) | Every setting `uap config` exposes — what it does, default, and a recommendation (generated from the registry); `uap config list/explain/set/doctor/wizard` ⭐ |
|
|
93
97
|
| [Database Schema](reference/DATABASE_SCHEMA.md) | SQLite databases + Qdrant collections |
|
|
@@ -132,6 +136,7 @@ the decision record.
|
|
|
132
136
|
| [Harness engineering uplift](plans/harness-engineering-uplift-2026-07-31.md) | Three-paper analysis of harness variance and what UAP does about it |
|
|
133
137
|
| [K3 UAP uplift validation](plans/k3-uap-uplift-validation.md) | Plan to measure UAP's paired uplift on Kimi K3 by re-running Terminal-Bench 2.1 with Factory Droid as the harness, UAP on vs off |
|
|
134
138
|
| [Qwen3.8 UAP re-test plan](plans/qwen38-uap-retest-plan.md) | Plan to re-run the paired benchmark matrix on the current local Qwen3.8-27B stack with UAP v1.224.x, replicating the +20pp gate-loop finding and checking the hardened deliver pipeline |
|
|
139
|
+
| [System-1 uplift plan](plans/system1-uplift-2026-09-18.md) | Approved uplift from the 2026-09-18 ten-source analysis (Jev/Foreman/Fluent/CEK et al): Wave 0 quick wins + Wave 1 System-1 decision layer, local-only classifier trained from UAP telemetry |
|
|
135
140
|
|
|
136
141
|
## Specs & performance
|
|
137
142
|
|
|
@@ -140,6 +145,7 @@ the decision record.
|
|
|
140
145
|
| [P0 anti-vacuous spec](specs/p0-anti-vacuous.md) | The verbatim-operations spec behind deliver's anti-stub hardening |
|
|
141
146
|
| [Dashboard uplift spec](DASHBOARD_UPLIFT_SPEC.md) | Replacing the monolithic dashboard with the modular token-locked console |
|
|
142
147
|
| [Performance baseline](performance/baseline-2026-03-27.json) | Machine-recorded perf baseline (heap, query latency) |
|
|
148
|
+
| [Reliability ladder](performance/reliability-ladder.md) | Measured technique-selection table for the decision loop; `npm run bench:ladder` regenerates it from `benchmark-results/` |
|
|
143
149
|
|
|
144
150
|
## Contributing
|
|
145
151
|
|
|
@@ -0,0 +1,75 @@
|
|
|
1
|
+
# ADR 0001: AutoMode risk classification ships as a Python heuristic floor ahead of the TS SYS1 head
|
|
2
|
+
|
|
3
|
+
- Status: Accepted
|
|
4
|
+
- Date: 2026-07-13 (uplift 1.3, branch `feature/304-automode-risk`)
|
|
5
|
+
|
|
6
|
+
## Context
|
|
7
|
+
|
|
8
|
+
AutoMode needs risk classification of tool calls **before dispatch** —
|
|
9
|
+
ahead of the verb/regex heuristics in the client-side enforcer chain
|
|
10
|
+
(`src/policies/enforcers/*.py`). The designed classifier is the TypeScript
|
|
11
|
+
SYS1 head (`src/classify/`, PR #807, branch `feature/301-sys1-classifier`),
|
|
12
|
+
but that branch is **unmerged and unreachable from Python**: the proxy
|
|
13
|
+
(`tools/agents/scripts/anthropic_proxy.py`) and the enforcers are Python,
|
|
14
|
+
and there is no TS→Python bridge in the serving path. Blocking AutoMode on
|
|
15
|
+
the TS merge would leave the proxy/enforcer stack with no risk signal at
|
|
16
|
+
all.
|
|
17
|
+
|
|
18
|
+
Two further constraints shaped the decision:
|
|
19
|
+
|
|
20
|
+
1. The deterministic enforcers are the blocking floor and must stay exactly
|
|
21
|
+
as strict — any classifier output is **advisory only**.
|
|
22
|
+
2. A wrong signal is asymmetric: a *missed* destructive call is the
|
|
23
|
+
unacceptable error; a false positive costs one advisory log line.
|
|
24
|
+
|
|
25
|
+
## Decision
|
|
26
|
+
|
|
27
|
+
Ship a **Python-resident calibrated heuristic scorer**
|
|
28
|
+
(`tools/agents/scripts/tool_risk.py`) behind a stable contract,
|
|
29
|
+
`classify_tool_call(tool_name, tool_input) -> RiskAssessment`, wired
|
|
30
|
+
advisory-only into the proxy's three outbound tool_use interception points
|
|
31
|
+
(debug log per call; `dashboard_events` telemetry for score ≥ 4; never
|
|
32
|
+
blocks, never mutates the stream; `PROXY_AUTOMODE_RISK=off` hatch).
|
|
33
|
+
|
|
34
|
+
- **Vocabulary inspired by, not ported from, the TS baseline.** The SYS1
|
|
35
|
+
`baseline.ts` scores prose state descriptions; this scorer parses shell
|
|
36
|
+
structure (verbs, flags, redirects, launchers). The domains differ, so we
|
|
37
|
+
do not claim cross-stack parity.
|
|
38
|
+
- **Deliberate duplication against `enforcement_self_protect.py`.** The
|
|
39
|
+
scorer re-implements launcher step-over, find-mutating flags, and the
|
|
40
|
+
sed `-i` conditional rather than importing the enforcer, because the
|
|
41
|
+
enforcer is a blocking gate with repo-root side effects and the scorer
|
|
42
|
+
must stay pure and importable from the proxy hot path. The mitigation for
|
|
43
|
+
the copy-divergence risk is a **drift-guard test**
|
|
44
|
+
(`EnforcerParityDriftGuardTest` in `test_tool_risk.py`) that imports both
|
|
45
|
+
modules and asserts the launcher sets, find-mutating semantics, and sed
|
|
46
|
+
`-i` semantics agree — divergence becomes a CI failure, not silent drift.
|
|
47
|
+
- **Deferred consumption.** For now the classification is consumed only by
|
|
48
|
+
logs and telemetry. The follow-up that produces *production* calibration
|
|
49
|
+
data is enforcer consumption plus a shadow-disagreement signal: run the
|
|
50
|
+
scorer beside the enforcers, record where advisory and floor disagree,
|
|
51
|
+
and let that corpus — not hand-labeled fixtures alone — calibrate the
|
|
52
|
+
promoted head.
|
|
53
|
+
- **Real promotion path.** The SYS1 head, when it lands, is a
|
|
54
|
+
Python-resident model (or a local sidecar) behind the same
|
|
55
|
+
`classify_tool_call` contract, gated by
|
|
56
|
+
`tools/agents/scripts/tool_risk_calibrate.py` at held-out destructive
|
|
57
|
+
recall 1.0 on a corpus that by then includes production disagreement
|
|
58
|
+
data. Wiring, harness, and doctrine do not change.
|
|
59
|
+
|
|
60
|
+
## Consequences
|
|
61
|
+
|
|
62
|
+
- The proxy/enforcer stack gets AutoMode risk classification now, with
|
|
63
|
+
measured latency (p95 ≈ 0.03 ms over 10k calls) far under the 50 ms
|
|
64
|
+
budget reserved for a future model call.
|
|
65
|
+
- The advisory path fails **open** (classifier exception → warning +
|
|
66
|
+
pass-through) precisely because the enforcer floor fails **closed**; a
|
|
67
|
+
dead scorer degrades to today's behavior plus one startup warning if the
|
|
68
|
+
import itself failed.
|
|
69
|
+
- The heuristic floor's perfect fixture numbers are a *contrast baseline*
|
|
70
|
+
on a small, hand-labeled probe (held-out n=4–10 per class), explicitly
|
|
71
|
+
caveated in `docs/performance/automode-risk-calibration.md`; they are not
|
|
72
|
+
evidence of model-grade generalization.
|
|
73
|
+
- The duplication debt against `enforcement_self_protect.py` is real but
|
|
74
|
+
bounded and guarded; if the enforcer grows a new evasion shape, the
|
|
75
|
+
drift-guard test forces the scorer to follow.
|
|
@@ -0,0 +1,81 @@
|
|
|
1
|
+
# ADR-0002: Evidence-bound merge candidates (gate evidence)
|
|
2
|
+
|
|
3
|
+
- **Status**: Accepted
|
|
4
|
+
- **Date**: 2026-09-18
|
|
5
|
+
- **Policy**: `evidence-bound-ship` (`src/policies/schemas/policies/evidence-bound-ship.md`)
|
|
6
|
+
- **Enforcer**: `src/policies/enforcers/evidence_bound_ship.py`
|
|
7
|
+
- **Writer**: `src/delivery/gate-evidence.ts` (emit seam: `src/cli/deliver.ts`)
|
|
8
|
+
- **Reference**: `docs/reference/evidence-bound-ship.md`
|
|
9
|
+
|
|
10
|
+
## Context
|
|
11
|
+
|
|
12
|
+
Deliver missions claim "all gates pass", but nothing bound that claim to the
|
|
13
|
+
commit a ship action actually publishes. An agent could run the gates on one
|
|
14
|
+
tree, then amend, swap, or dirty the commit and ship code the gates never saw —
|
|
15
|
+
the existing ship gates (`expert-review-required`) trusted a *review verdict*,
|
|
16
|
+
not machine proof tied to a SHA. Uplift 1.4 requires merge candidates to carry
|
|
17
|
+
their gate evidence.
|
|
18
|
+
|
|
19
|
+
Two properties make such evidence worth anything, and both shaped the design:
|
|
20
|
+
|
|
21
|
+
1. **The agent must not be able to write it.** A `.uap/evidence/<sha>.json`
|
|
22
|
+
the agent can forge is worse than none — it converts "no proof" into
|
|
23
|
+
"plausible proof". `.uap/evidence/` is therefore a self-protect
|
|
24
|
+
PROTECTED_TARGET with no carve-out, and the interpreter-write rule
|
|
25
|
+
(`python3 -c`, `node -e`, heredoc) covers the directory, because the
|
|
26
|
+
verb-based Bash scan cannot see an interpreter-mediated write.
|
|
27
|
+
2. **The evidence must name the exact commit it proves.** Binding to "the
|
|
28
|
+
branch" or "whenever the gates ran" re-admits the swap. So the artifact
|
|
29
|
+
records `candidateSha = HEAD` and is only written against a **clean** tree —
|
|
30
|
+
with uncommitted changes the gates provably ran against HEAD-plus-delta,
|
|
31
|
+
and the claim would be unprovable.
|
|
32
|
+
|
|
33
|
+
## Decision
|
|
34
|
+
|
|
35
|
+
- The deliver CLI records `.uap/evidence/<head-sha>.json` at the single
|
|
36
|
+
completion seam all runner kinds converge on, from the **last passing**
|
|
37
|
+
iteration's rung results (or the baseline ladder run when the run
|
|
38
|
+
short-circuits as `alreadyDelivered` — otherwise the supported commit →
|
|
39
|
+
deliver → ship flow dead-ends on an already-green tree).
|
|
40
|
+
- The ship enforcer **fails closed** on missing, oversized (>512KB),
|
|
41
|
+
malformed, wrong-version, SHA-mismatched, backdated, future-dated
|
|
42
|
+
(>5min skew), stale (default 24h), or gate-empty evidence; an unresolvable
|
|
43
|
+
commit time also blocks. The only fail-open is a tree that is not a git
|
|
44
|
+
repository.
|
|
45
|
+
- Ship-action detection is **shared** with `expert-review-required` (imported
|
|
46
|
+
`is_ship_action`) so the two gates can never disagree on what a ship is.
|
|
47
|
+
- Operator hatches are environment-only (`UAP_EVIDENCE_GATE_OFF=1`,
|
|
48
|
+
`UAP_EVIDENCE_MAX_AGE_HOURS`); the agent cannot set either inline or
|
|
49
|
+
persistently — enforcement-self-protect refuses the assignment text.
|
|
50
|
+
|
|
51
|
+
### Fail-soft emit / fail-closed gate asymmetry
|
|
52
|
+
|
|
53
|
+
Evidence **emission** never breaks a mission (a warning is the whole failure
|
|
54
|
+
mode): a deliver run must not fail because bookkeeping failed. The **gate**
|
|
55
|
+
fails closed: a ship without verifiable evidence is refused. The asymmetry has
|
|
56
|
+
a real recovery cost — when emission silently fails (dirty tree, unborn HEAD),
|
|
57
|
+
the operator's only feedback is a warning line, and the ship block arrives
|
|
58
|
+
later, detached from its cause. We accept this because the block message names
|
|
59
|
+
the fix (`commit, then re-run uap deliver`) and the escape hatch exists for
|
|
60
|
+
genuinely blocked operators. The alternative (fail-hard emission) would wedge
|
|
61
|
+
missions on a provenance feature; the other alternative (fail-open gate)
|
|
62
|
+
deletes the control.
|
|
63
|
+
|
|
64
|
+
## Consequences
|
|
65
|
+
|
|
66
|
+
- The supported flow is **commit → `uap deliver` → ship**. A deliver run that
|
|
67
|
+
leaves the tree dirty emits no evidence (warns only); the subsequent ship is
|
|
68
|
+
blocked until the candidate is committed and deliver re-run.
|
|
69
|
+
- **Local-only perimeter**: the gate fires only for ship actions routed
|
|
70
|
+
through this machine's policy chain. GitHub-web merges and machines without
|
|
71
|
+
the chain bypass it; squash/rebase merges mint a remote SHA no local
|
|
72
|
+
evidence can bind. Branch protection / required status checks remain the
|
|
73
|
+
remote-side complement (see the reference doc's Scope & limitations).
|
|
74
|
+
- `gh pr merge <N>` binds the **local** HEAD (no network/`gh` in the policy
|
|
75
|
+
chain); shipping from a checkout diverged from the PR head validates the
|
|
76
|
+
wrong commit and is blocked by design.
|
|
77
|
+
- Per-gate timestamps are reconstructed from rung durations, not measured —
|
|
78
|
+
honest ordering, approximate clock; documented in the schema.
|
|
79
|
+
- The artifact carries a `hatches` audit field (which gate-affecting env vars
|
|
80
|
+
were set at deliver time) so a reviewer can see the gates ran with e.g. the
|
|
81
|
+
quality gate disabled; the enforcer deliberately does not act on it.
|
|
@@ -0,0 +1,98 @@
|
|
|
1
|
+
# Capacity Policy as Code
|
|
2
|
+
|
|
3
|
+
Uplift 0.5 of the [System-1 uplift plan](../plans/system1-uplift-2026-09-18.md):
|
|
4
|
+
every UAP-managed service declares its resource budget, its headroom
|
|
5
|
+
requirement, and its restart discipline; `uap doctor` probes each service and
|
|
6
|
+
reports GREEN/RED/DARK health against the declaration.
|
|
7
|
+
|
|
8
|
+
This generalizes the 2026-09-18 llama.cpp OOM fix: four SIGABRT crashes
|
|
9
|
+
happened because a KV-cache budget was resolved against a GPU that also runs
|
|
10
|
+
a desktop, and nothing declared that relationship. The fix held, but the
|
|
11
|
+
*discipline* (declare budget + headroom + restart baseline, watch for
|
|
12
|
+
violations) lived in a commit message. Now it is data.
|
|
13
|
+
|
|
14
|
+
## Policy file
|
|
15
|
+
|
|
16
|
+
Resolution order: `--policy <path>`, then `config/capacity-policy.json` in
|
|
17
|
+
the project, then `~/.config/uap/capacity-policy.json`.
|
|
18
|
+
|
|
19
|
+
```json
|
|
20
|
+
{
|
|
21
|
+
"version": 1,
|
|
22
|
+
"services": [
|
|
23
|
+
{
|
|
24
|
+
"name": "gsq-rco-server",
|
|
25
|
+
"systemd": { "unit": "uap-gsq-rco-server.service", "scope": "user" },
|
|
26
|
+
"budget": { "vramMiB": 5120, "note": "--vbr-vram 5120M (OOM fix 2026-09-18)" },
|
|
27
|
+
"headroom": { "gpuMinFreeMiB": 600 },
|
|
28
|
+
"restartBudget": { "knownRestarts": 0, "allowedNew": 0 }
|
|
29
|
+
}
|
|
30
|
+
]
|
|
31
|
+
}
|
|
32
|
+
```
|
|
33
|
+
|
|
34
|
+
- **budget** — what the service is configured to stay inside (free-text
|
|
35
|
+
`note` records the provenance: which incident, which flag).
|
|
36
|
+
- `budget.vramMiB` declares the service's *own* allocator limit (e.g.
|
|
37
|
+
`--vbr-vram 5120M`), which the server enforces internally. The doctor
|
|
38
|
+
does **not** compare it against the process's total GPU footprint —
|
|
39
|
+
the host-side invariant it owns is `headroom.gpuMinFreeMiB`.
|
|
40
|
+
- `budget.rssMiB` **is** enforced: compared against the unit's
|
|
41
|
+
`MemoryCurrent`. Declaring it when systemd cannot report memory
|
|
42
|
+
(cgroup accounting off) is RED — a declared budget that cannot be
|
|
43
|
+
verified is not healthy.
|
|
44
|
+
- **headroom.gpuMinFreeMiB** — free VRAM the *host* must keep; below it the
|
|
45
|
+
service is in OOM territory even while running.
|
|
46
|
+
- **restartBudget** — `knownRestarts` is the accepted historical baseline
|
|
47
|
+
(e.g. crashes already fixed); `allowedNew` new crashes beyond that turn the
|
|
48
|
+
service RED. Zero is the right default after a fix lands.
|
|
49
|
+
|
|
50
|
+
## Health semantics
|
|
51
|
+
|
|
52
|
+
| Health | Meaning |
|
|
53
|
+
| --- | --- |
|
|
54
|
+
| GREEN | Unit active and serving, restarts within budget, headroom verified. |
|
|
55
|
+
| RED | A budget is violated or unverifiable: new crashes since the baseline, GPU free below declared headroom, RSS over budget, headroom/RSS declared but not probeable. Also RED while `activating` (e.g. `SubState=auto-restart`) — a `Restart=always` crash loop is not yet serving, so it never reads GREEN, with or without a declared `restartBudget`. |
|
|
56
|
+
| DARK | Unit failed/inactive/unresolvable — the service is *absent*, not degraded. |
|
|
57
|
+
| UNKNOWN | systemd itself unavailable (container, macOS) — never a fabricated GREEN. |
|
|
58
|
+
|
|
59
|
+
## Running it
|
|
60
|
+
|
|
61
|
+
```bash
|
|
62
|
+
uap doctor # report, advisory (exit 0)
|
|
63
|
+
uap doctor --json # machine-readable
|
|
64
|
+
uap doctor --strict # exit 1 on RED/DARK — for CI and monitor loops
|
|
65
|
+
```
|
|
66
|
+
|
|
67
|
+
Reference implementation: `config/capacity-policy.json` in this repo is the
|
|
68
|
+
**live policy of the gsq build host**, committed deliberately — the uplift
|
|
69
|
+
requires the gsq-rco llama.cpp server and its watchdog monitor to be the
|
|
70
|
+
reference implementations, and keeping the real file in-repo means `uap
|
|
71
|
+
doctor` run from this checkout reports that host truthfully. Consumers
|
|
72
|
+
elsewhere should ship their own `config/capacity-policy.json` (project) or
|
|
73
|
+
`~/.config/uap/capacity-policy.json` (operator); the resolution order above
|
|
74
|
+
makes the committed file the fallback only inside this repo.
|
|
75
|
+
|
|
76
|
+
`uap doctor --strict` is the natural health source for the monitor service
|
|
77
|
+
to consult before restarting anything, and for CI to refuse a deploy into a
|
|
78
|
+
degraded host.
|
|
79
|
+
|
|
80
|
+
## Decisions
|
|
81
|
+
|
|
82
|
+
- **Restarts are budgeted against a baseline, not a rate.** `NRestarts` is a
|
|
83
|
+
monotonic counter; "crashes per day" windows go stale. The policy records
|
|
84
|
+
how many restarts are *explained* (pre-fix) and alerts on new ones.
|
|
85
|
+
Known hole: `systemctl restart` / `reset-failed` resets `NRestarts` to 0,
|
|
86
|
+
so bouncing a crash-looping unit produces a false GREEN until the next
|
|
87
|
+
crash. Re-baseline `knownRestarts` in the same commit as any legitimate
|
|
88
|
+
restart (deploy, config change) — a `uap doctor --rebaseline` verb is
|
|
89
|
+
planned as a follow-up; until it lands the edit is manual and deliberate.
|
|
90
|
+
- **UNKNOWN instead of fail-open GREEN.** A doctor that cannot probe must say
|
|
91
|
+
so; a green report from a blind probe is the failure mode this tool exists
|
|
92
|
+
to prevent.
|
|
93
|
+
- **One GPU probe per run** shared across services — nvidia-smi is not cheap.
|
|
94
|
+
- **Probes fail soft on a 5s timeout.** A wedged driver or D-Bus makes the
|
|
95
|
+
doctor report UNKNOWN/RED-unverified instead of hanging the monitor loop
|
|
96
|
+
that called it.
|
|
97
|
+
- **`uap doctor` vs `uap config doctor`:** the latter diagnoses *settings*
|
|
98
|
+
health; this diagnoses *running services* against declared capacity.
|