@miller-tech/uap 2.21.3 → 2.36.4
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +22 -10
- package/config/model-profiles/qwen38-coder390.json +46 -0
- package/config/model-profiles/qwen38.json +3 -2
- package/config/model-registry.json +110 -0
- package/dist/.tsbuildinfo +1 -1
- package/dist/benchmarks/paired/types.d.ts +2 -2
- package/dist/bin/cli.js +315 -20
- package/dist/bin/cli.js.map +1 -1
- package/dist/cli/attention.d.ts +6 -0
- package/dist/cli/attention.d.ts.map +1 -0
- package/dist/cli/attention.js +115 -0
- package/dist/cli/attention.js.map +1 -0
- package/dist/cli/bench-report.d.ts +82 -0
- package/dist/cli/bench-report.d.ts.map +1 -0
- package/dist/cli/bench-report.js +237 -0
- package/dist/cli/bench-report.js.map +1 -0
- package/dist/cli/clients.d.ts +6 -0
- package/dist/cli/clients.d.ts.map +1 -0
- package/dist/cli/clients.js +142 -0
- package/dist/cli/clients.js.map +1 -0
- package/dist/cli/dashboard-grants.d.ts +26 -0
- package/dist/cli/dashboard-grants.d.ts.map +1 -0
- package/dist/cli/dashboard-grants.js +69 -0
- package/dist/cli/dashboard-grants.js.map +1 -0
- package/dist/cli/dashboard-pace.d.ts +31 -0
- package/dist/cli/dashboard-pace.d.ts.map +1 -0
- package/dist/cli/dashboard-pace.js +42 -0
- package/dist/cli/dashboard-pace.js.map +1 -0
- package/dist/cli/dashboard-session.d.ts +25 -0
- package/dist/cli/dashboard-session.d.ts.map +1 -0
- package/dist/cli/dashboard-session.js +814 -0
- package/dist/cli/dashboard-session.js.map +1 -0
- package/dist/cli/dashboard-types.d.ts +18 -0
- package/dist/cli/dashboard-types.d.ts.map +1 -0
- package/dist/cli/dashboard-types.js +9 -0
- package/dist/cli/dashboard-types.js.map +1 -0
- package/dist/cli/dashboard.d.ts +2 -18
- package/dist/cli/dashboard.d.ts.map +1 -1
- package/dist/cli/dashboard.js +43 -793
- package/dist/cli/dashboard.js.map +1 -1
- package/dist/cli/deliver.d.ts +35 -0
- package/dist/cli/deliver.d.ts.map +1 -1
- package/dist/cli/deliver.js +93 -9
- package/dist/cli/deliver.js.map +1 -1
- package/dist/cli/diff.d.ts +85 -0
- package/dist/cli/diff.d.ts.map +1 -0
- package/dist/cli/diff.js +293 -0
- package/dist/cli/diff.js.map +1 -0
- package/dist/cli/eta.d.ts +35 -0
- package/dist/cli/eta.d.ts.map +1 -0
- package/dist/cli/eta.js +101 -0
- package/dist/cli/eta.js.map +1 -0
- package/dist/cli/handsfree.d.ts +7 -0
- package/dist/cli/handsfree.d.ts.map +1 -1
- package/dist/cli/handsfree.js +27 -0
- package/dist/cli/handsfree.js.map +1 -1
- package/dist/cli/hooks.d.ts.map +1 -1
- package/dist/cli/hooks.js +126 -5
- package/dist/cli/hooks.js.map +1 -1
- package/dist/cli/loops.d.ts +24 -5
- package/dist/cli/loops.d.ts.map +1 -1
- package/dist/cli/loops.js +92 -12
- package/dist/cli/loops.js.map +1 -1
- package/dist/cli/models.d.ts +62 -0
- package/dist/cli/models.d.ts.map +1 -0
- package/dist/cli/models.js +567 -0
- package/dist/cli/models.js.map +1 -0
- package/dist/cli/plan.d.ts +9 -0
- package/dist/cli/plan.d.ts.map +1 -1
- package/dist/cli/plan.js +57 -4
- package/dist/cli/plan.js.map +1 -1
- package/dist/cli/proxy-relay.d.ts +55 -0
- package/dist/cli/proxy-relay.d.ts.map +1 -0
- package/dist/cli/proxy-relay.js +116 -0
- package/dist/cli/proxy-relay.js.map +1 -0
- package/dist/cli/proxy.d.ts +4 -0
- package/dist/cli/proxy.d.ts.map +1 -1
- package/dist/cli/proxy.js +26 -1
- package/dist/cli/proxy.js.map +1 -1
- package/dist/cli/sandbox.d.ts +101 -4
- package/dist/cli/sandbox.d.ts.map +1 -1
- package/dist/cli/sandbox.js +393 -44
- package/dist/cli/sandbox.js.map +1 -1
- package/dist/cli/self.d.ts +13 -0
- package/dist/cli/self.d.ts.map +1 -0
- package/dist/cli/self.js +94 -0
- package/dist/cli/self.js.map +1 -0
- package/dist/cli/systemd-services.d.ts +2 -8
- package/dist/cli/systemd-services.d.ts.map +1 -1
- package/dist/cli/systemd-services.js +5 -20
- package/dist/cli/systemd-services.js.map +1 -1
- package/dist/cli/task.js +1 -1
- package/dist/cli/task.js.map +1 -1
- package/dist/cli/wizard-config.d.ts.map +1 -1
- package/dist/cli/wizard-config.js +31 -2
- package/dist/cli/wizard-config.js.map +1 -1
- package/dist/config/settings-registry.d.ts +1 -1
- package/dist/config/settings-registry.d.ts.map +1 -1
- package/dist/config/settings-registry.js +108 -0
- package/dist/config/settings-registry.js.map +1 -1
- package/dist/coordination/capability-router.js +2 -2
- package/dist/coordination/capability-router.js.map +1 -1
- package/dist/coordination/database.d.ts +13 -0
- package/dist/coordination/database.d.ts.map +1 -1
- package/dist/coordination/database.js +63 -7
- package/dist/coordination/database.js.map +1 -1
- package/dist/coordination/eta-inject.d.ts +32 -0
- package/dist/coordination/eta-inject.d.ts.map +1 -0
- package/dist/coordination/eta-inject.js +64 -0
- package/dist/coordination/eta-inject.js.map +1 -0
- package/dist/coordination/reactor.d.ts.map +1 -1
- package/dist/coordination/reactor.js +14 -0
- package/dist/coordination/reactor.js.map +1 -1
- package/dist/coordination/service.d.ts +18 -8
- package/dist/coordination/service.d.ts.map +1 -1
- package/dist/coordination/service.js +38 -28
- package/dist/coordination/service.js.map +1 -1
- package/dist/dashboard/client-registry.d.ts +163 -0
- package/dist/dashboard/client-registry.d.ts.map +1 -0
- package/dist/dashboard/client-registry.js +549 -0
- package/dist/dashboard/client-registry.js.map +1 -0
- package/dist/dashboard/controls.d.ts.map +1 -1
- package/dist/dashboard/controls.js +33 -4
- package/dist/dashboard/controls.js.map +1 -1
- package/dist/dashboard/data-seeder.d.ts +10 -7
- package/dist/dashboard/data-seeder.d.ts.map +1 -1
- package/dist/dashboard/data-seeder.js +31 -75
- package/dist/dashboard/data-seeder.js.map +1 -1
- package/dist/dashboard/data-service.d.ts +9 -0
- package/dist/dashboard/data-service.d.ts.map +1 -1
- package/dist/dashboard/data-service.js +32 -3
- package/dist/dashboard/data-service.js.map +1 -1
- package/dist/dashboard/placement-routes.d.ts +119 -0
- package/dist/dashboard/placement-routes.d.ts.map +1 -0
- package/dist/dashboard/placement-routes.js +331 -0
- package/dist/dashboard/placement-routes.js.map +1 -0
- package/dist/dashboard/server.d.ts.map +1 -1
- package/dist/dashboard/server.js +408 -2
- package/dist/dashboard/server.js.map +1 -1
- package/dist/delivery/attention.d.ts +130 -0
- package/dist/delivery/attention.d.ts.map +1 -0
- package/dist/delivery/attention.js +341 -0
- package/dist/delivery/attention.js.map +1 -0
- package/dist/delivery/decompose.d.ts +18 -0
- package/dist/delivery/decompose.d.ts.map +1 -1
- package/dist/delivery/decompose.js +22 -1
- package/dist/delivery/decompose.js.map +1 -1
- package/dist/delivery/eta-estimate.d.ts +111 -0
- package/dist/delivery/eta-estimate.d.ts.map +1 -0
- package/dist/delivery/eta-estimate.js +171 -0
- package/dist/delivery/eta-estimate.js.map +1 -0
- package/dist/delivery/eta.d.ts +23 -0
- package/dist/delivery/eta.d.ts.map +1 -0
- package/dist/delivery/eta.js +59 -0
- package/dist/delivery/eta.js.map +1 -0
- package/dist/delivery/orchestrated-mission.d.ts +10 -0
- package/dist/delivery/orchestrated-mission.d.ts.map +1 -1
- package/dist/delivery/orchestrated-mission.js +15 -1
- package/dist/delivery/orchestrated-mission.js.map +1 -1
- package/dist/delivery/plan-check.d.ts +7 -1
- package/dist/delivery/plan-check.d.ts.map +1 -1
- package/dist/delivery/plan-check.js +23 -2
- package/dist/delivery/plan-check.js.map +1 -1
- package/dist/delivery/run-state.d.ts +1 -1
- package/dist/delivery/run-state.d.ts.map +1 -1
- package/dist/delivery/run-state.js +8 -1
- package/dist/delivery/run-state.js.map +1 -1
- package/dist/delivery/task-orchestrator.d.ts +8 -3
- package/dist/delivery/task-orchestrator.d.ts.map +1 -1
- package/dist/delivery/task-orchestrator.js +75 -43
- package/dist/delivery/task-orchestrator.js.map +1 -1
- package/dist/delivery/task-sync.d.ts +21 -0
- package/dist/delivery/task-sync.d.ts.map +1 -1
- package/dist/delivery/task-sync.js +44 -8
- package/dist/delivery/task-sync.js.map +1 -1
- package/dist/delivery/task-workspace.d.ts +3 -3
- package/dist/delivery/task-workspace.js +3 -3
- package/dist/generators/self-card.d.ts +65 -0
- package/dist/generators/self-card.d.ts.map +1 -0
- package/dist/generators/self-card.js +256 -0
- package/dist/generators/self-card.js.map +1 -0
- package/dist/inference/probe.js +1 -1
- package/dist/inference/probe.js.map +1 -1
- package/dist/memory/prepopulate.d.ts.map +1 -1
- package/dist/memory/prepopulate.js +2 -0
- package/dist/memory/prepopulate.js.map +1 -1
- package/dist/models/analytics.d.ts +32 -0
- package/dist/models/analytics.d.ts.map +1 -1
- package/dist/models/analytics.js +68 -5
- package/dist/models/analytics.js.map +1 -1
- package/dist/models/plan-readability.d.ts +55 -0
- package/dist/models/plan-readability.d.ts.map +1 -0
- package/dist/models/plan-readability.js +339 -0
- package/dist/models/plan-readability.js.map +1 -0
- package/dist/placement/admission.d.ts +59 -0
- package/dist/placement/admission.d.ts.map +1 -0
- package/dist/placement/admission.js +197 -0
- package/dist/placement/admission.js.map +1 -0
- package/dist/placement/auto.d.ts +44 -0
- package/dist/placement/auto.d.ts.map +1 -0
- package/dist/placement/auto.js +212 -0
- package/dist/placement/auto.js.map +1 -0
- package/dist/placement/enforce.d.ts +128 -0
- package/dist/placement/enforce.d.ts.map +1 -0
- package/dist/placement/enforce.js +655 -0
- package/dist/placement/enforce.js.map +1 -0
- package/dist/placement/idle.d.ts +50 -0
- package/dist/placement/idle.d.ts.map +1 -0
- package/dist/placement/idle.js +213 -0
- package/dist/placement/idle.js.map +1 -0
- package/dist/placement/ledger.d.ts +323 -0
- package/dist/placement/ledger.d.ts.map +1 -0
- package/dist/placement/ledger.js +263 -0
- package/dist/placement/ledger.js.map +1 -0
- package/dist/placement/preview.d.ts +51 -0
- package/dist/placement/preview.d.ts.map +1 -0
- package/dist/placement/preview.js +61 -0
- package/dist/placement/preview.js.map +1 -0
- package/dist/placement/probes.d.ts +47 -0
- package/dist/placement/probes.d.ts.map +1 -0
- package/dist/placement/probes.js +207 -0
- package/dist/placement/probes.js.map +1 -0
- package/dist/placement/proxy-env.d.ts +21 -0
- package/dist/placement/proxy-env.d.ts.map +1 -0
- package/dist/placement/proxy-env.js +42 -0
- package/dist/placement/proxy-env.js.map +1 -0
- package/dist/placement/registry.d.ts +1038 -0
- package/dist/placement/registry.d.ts.map +1 -0
- package/dist/placement/registry.js +274 -0
- package/dist/placement/registry.js.map +1 -0
- package/dist/policies/liveness.d.ts +6 -8
- package/dist/policies/liveness.d.ts.map +1 -1
- package/dist/policies/liveness.js +6 -25
- package/dist/policies/liveness.js.map +1 -1
- package/dist/policies/sandbox-anchor.d.ts +91 -0
- package/dist/policies/sandbox-anchor.d.ts.map +1 -0
- package/dist/policies/sandbox-anchor.js +187 -0
- package/dist/policies/sandbox-anchor.js.map +1 -0
- package/dist/policies/sandbox-grants.d.ts +131 -0
- package/dist/policies/sandbox-grants.d.ts.map +1 -0
- package/dist/policies/sandbox-grants.js +562 -0
- package/dist/policies/sandbox-grants.js.map +1 -0
- package/dist/state/manifest.d.ts +4 -2
- package/dist/state/manifest.d.ts.map +1 -1
- package/dist/state/manifest.js +32 -16
- package/dist/state/manifest.js.map +1 -1
- package/dist/utils/checkout-root.d.ts +2 -0
- package/dist/utils/checkout-root.d.ts.map +1 -0
- package/dist/utils/checkout-root.js +43 -0
- package/dist/utils/checkout-root.js.map +1 -0
- package/dist/utils/resolve-uap-bin.d.ts +5 -0
- package/dist/utils/resolve-uap-bin.d.ts.map +1 -0
- package/dist/utils/resolve-uap-bin.js +33 -0
- package/dist/utils/resolve-uap-bin.js.map +1 -0
- package/dist/utils/terminal-safe.d.ts +12 -0
- package/dist/utils/terminal-safe.d.ts.map +1 -0
- package/dist/utils/terminal-safe.js +53 -0
- package/dist/utils/terminal-safe.js.map +1 -0
- package/docs/ATTRIBUTION.md +1 -0
- package/docs/DASHBOARD_UPLIFT_SPEC.md +8 -8
- package/docs/INDEX.md +6 -5
- package/docs/architecture/OVERVIEW.md +15 -3
- package/docs/architecture/PROTOCOL.md +7 -1
- package/docs/architecture/adr/0006-operator-directed-model-placement.md +86 -0
- package/docs/architecture/adr/0007-auto-displacement-allowlist-as-standing-consent.md +79 -0
- package/docs/architecture/adr/0008-fleet-clients-link-out.md +92 -0
- package/docs/architecture/adr/0009-completion-triggered-dispatch-and-per-leaf-model-dials.md +71 -0
- package/docs/architecture/adr/0010-attention-queue-and-afk-auto-decider.md +159 -0
- package/docs/architecture/adr/0011-sandbox-grants-and-network-egress-boundary.md +219 -0
- package/docs/architecture/adr/0012-operator-legibility-surfaces.md +175 -0
- package/docs/getting-started/INSTALLATION.md +1 -1
- package/docs/guides/AUTOMATIC_FEATURES.md +8 -0
- package/docs/guides/LOCAL_MODELS.md +1 -1
- package/docs/guides/PROXY.md +2 -1
- package/docs/guides/SANDBOX.md +91 -4
- package/docs/integrations/MCP_ROUTER.md +1 -1
- package/docs/plans/model-performance-proof-options.md +176 -0
- package/docs/reference/API.md +1 -1
- package/docs/reference/CLI.md +169 -4
- package/docs/reference/COMPARISON.md +1 -1
- package/docs/reference/CONFIGURATION.md +12 -3
- package/docs/reference/CONFIGURATION_REFERENCE.md +269 -14
- package/docs/reference/DATABASE_SCHEMA.md +1 -1
- package/docs/reference/FEATURES.md +15 -3
- package/docs/reference/PATTERNS.md +1 -1
- package/docs/reference/PLATFORMS.md +14 -1
- package/docs/specs/operator-model-placement.md +1314 -0
- package/docs/specs/operator-model-placement.review.md +339 -0
- package/package.json +2 -2
- package/scripts/maintenance/publish-npm.sh +23 -11
- package/scripts/version-bump.sh +6 -51
- package/src/policies/enforcers/__pycache__/_common.cpython-311.pyc +0 -0
- package/src/policies/enforcers/__pycache__/expert_review_required.cpython-311.pyc +0 -0
- package/src/policies/enforcers/__pycache__/quality_metrics_gate.cpython-311.pyc +0 -0
- package/src/policies/schemas/policies/mandatory-testing-deployment.md +2 -5
- package/templates/hooks/__pycache__/deliver_autoroute.cpython-311.pyc +0 -0
- package/templates/hooks/session-start.sh +44 -0
- package/tools/agents/UAP/cli.py +14 -3
- package/tools/agents/UAP/paths.py +50 -0
- package/tools/agents/scripts/__pycache__/toolcall_path_normalizer.cpython-311.pyc +0 -0
- package/tools/agents/scripts/anthropic_proxy.py +1420 -100
- package/tools/agents/scripts/toolcall_path_normalizer.py +46 -1
- package/tools/agents/tests/test_anthropic_proxy_streaming.py +29 -6
- package/tools/agents/tests/test_doubling_break.py +64 -5
- package/tools/agents/tests/test_error_loop_break.py +10 -4
- package/tools/agents/tests/test_error_loop_ignores_correctives.py +13 -2
- package/tools/agents/tests/test_lockstep_escalation.py +280 -9
- package/tools/agents/tests/test_path_containment.py +94 -0
- package/tools/agents/tests/test_proxy_unix_socket.py +162 -0
- package/tools/agents/tests/test_signature_quality.py +400 -0
- package/tools/agents/tests/test_uap_compliance.py +11 -1
- package/web/dash/core.js +1 -0
- package/web/dash/tab-clients.js +196 -0
- package/web/dash/tab-models.js +387 -8
- package/web/dash/tab-policies.js +87 -4
- package/web/dash/tabs.js +77 -73
- package/web/dashboard.html +6 -0
- package/src/policies/enforcers/__pycache__/_common.cpython-312.pyc +0 -0
- package/src/policies/enforcers/__pycache__/expert_review_required.cpython-312.pyc +0 -0
- package/src/policies/enforcers/__pycache__/quality_metrics_gate.cpython-312.pyc +0 -0
- package/templates/hooks/__pycache__/deliver_autoroute.cpython-312.pyc +0 -0
- package/tools/agents/scripts/__pycache__/toolcall_path_normalizer.cpython-312.pyc +0 -0
- package/web/dash/tab-agents.js +0 -21
- package/web/dash/tab-deliver.js +0 -12
- package/web/dash/tab-memory.js +0 -12
- package/web/dash/tab-orchestration.js +0 -21
- package/web/dash/tab-overview.js +0 -21
- package/web/dash/tab-tasks.js +0 -21
|
@@ -0,0 +1,175 @@
|
|
|
1
|
+
# ADR 0012: Operator legibility surfaces — measured pace, local diff view, bench report reader
|
|
2
|
+
|
|
3
|
+
- Status: Accepted
|
|
4
|
+
- Date: 2026-10-11
|
|
5
|
+
- Spec: the operator's own legibility (S3 local diff view, S1b ETA
|
|
6
|
+
surfaces + paired bench delta report).
|
|
7
|
+
- Supersedes: none — these are read-only surfaces over telemetry that
|
|
8
|
+
already existed (the ETA sample stream, git, the paired-bench
|
|
9
|
+
reports); no schema or authority changes.
|
|
10
|
+
|
|
11
|
+
## Context
|
|
12
|
+
|
|
13
|
+
The agent-facing surfaces made the *agent* legible — to
|
|
14
|
+
itself (the self-knowledge card), to the plan, to the operator's consent
|
|
15
|
+
(grants, attention). What never got a surface was the operator's own three
|
|
16
|
+
questions, asked every day on a busy checkout:
|
|
17
|
+
|
|
18
|
+
1. **"What did the agent actually change around here?"** The answer lives
|
|
19
|
+
in git, but the honest answer spans three views at once — uncommitted
|
|
20
|
+
rows with per-file deltas, untracked files, and unpushed commits against
|
|
21
|
+
a base — and assembling it by hand is exactly the kind of repetitive
|
|
22
|
+
synthesis operators skip, then regret.
|
|
23
|
+
2. **"How long does this model actually take on this project?"** The ETA
|
|
24
|
+
sample stream (`delivery/eta.ts` → `model_analytics.db`, keyed
|
|
25
|
+
model × taskType × complexity) was recorded but nothing *read* it — so the
|
|
26
|
+
operator (and the agent) kept guessing pace from vibes, which is the
|
|
27
|
+
failure mode the whole program exists to end.
|
|
28
|
+
3. **"Did the measured delta hold?"** `uap bench paired` writes a full
|
|
29
|
+
AnalysisReport under `benchmark-results/paired-*/` and nobody ever looks
|
|
30
|
+
at it again; run-to-run drift (a delta that narrowed, a verdict that
|
|
31
|
+
flipped) was invisible unless someone diffed JSON by hand.
|
|
32
|
+
|
|
33
|
+
The bar (unchanged from the reliability ladder): **a number without its
|
|
34
|
+
provenance is noise.** Every surface below is built so it *cannot* print a
|
|
35
|
+
confident guess — thin data says so, unusable runs say so, unknown deltas
|
|
36
|
+
render as `±`, never as `0`.
|
|
37
|
+
|
|
38
|
+
## Decision
|
|
39
|
+
|
|
40
|
+
### S1b — the EWMA estimator (`src/delivery/eta-estimate.ts`) and its four surfaces
|
|
41
|
+
|
|
42
|
+
`ewmaEstimate(samples)` is pure math over a plain array — no store, no
|
|
43
|
+
fs, no clock — so the estimator is unit-testable and deterministic. It
|
|
44
|
+
folds the sample stream **oldest→newest** with a half-life **in samples**
|
|
45
|
+
(default 10, from `1 − 2^(−1/h)`), because the model, the harness, and the
|
|
46
|
+
task mix all drift; recency is the point. The honesty rules are load-bearing
|
|
47
|
+
and pinned by tests:
|
|
48
|
+
|
|
49
|
+
- **Thin data is not an estimate.** Fewer than 3 successful attempts in the
|
|
50
|
+
window yields `null` — surfaces show nothing rather than a number
|
|
51
|
+
invented from noise. 3–5 is `thin` (shown *with a caveat*, never
|
|
52
|
+
silently); ≥6 is `solid`.
|
|
53
|
+
- **The spread is part of the number.** min/max of the folded successes
|
|
54
|
+
travel with the estimate; a surface printing "~2 min" without the spread
|
|
55
|
+
is lying by omission.
|
|
56
|
+
- **Failures are visible.** The fold is over SUCCESS durations (that is
|
|
57
|
+
what "how long does a completing attempt take" means), but the estimate
|
|
58
|
+
also carries `attemptsPerSuccess` — and every wall-clock projection
|
|
59
|
+
(`remainingWallMs`, the handsfree line) scales by it, so a 50%-failure
|
|
60
|
+
key never presents its successes as its pace.
|
|
61
|
+
|
|
62
|
+
`estimatePace` is the fail-soft store wrapper (a broken store costs one
|
|
63
|
+
try/catch, never a crash in a surface — same contract as the S1a
|
|
64
|
+
recorder). The four surfaces:
|
|
65
|
+
|
|
66
|
+
- **`uap eta [--model|--taskType|--complexity|--all|--limit|--half-life|--json]`**
|
|
67
|
+
(`src/cli/eta.ts`) — one key, or `--all` for every key with history.
|
|
68
|
+
`--all` *names the too-thin keys* instead of silently omitting them, so
|
|
69
|
+
the operator knows why a key they expect is missing and what would fill
|
|
70
|
+
it.
|
|
71
|
+
- **handsfree status** — a remaining-work wall clock for the open ledger
|
|
72
|
+
items (`pending + failed`), scaled by attempts-per-success, shown only
|
|
73
|
+
when an estimate exists.
|
|
74
|
+
- **Dashboard models view** — a "Measured Pace" section (top 8 keys by
|
|
75
|
+
attempts): per-task EWMA, spread, n, success rate.
|
|
76
|
+
- **Reactor injection** (`src/coordination/eta-inject.ts`, keyed
|
|
77
|
+
`eta:pace`) — the agent's OWN pace as calibrated planning context: fires
|
|
78
|
+
only on solid estimates, because a thin-data guess would inject exactly
|
|
79
|
+
the confident noise the block exists to replace.
|
|
80
|
+
|
|
81
|
+
### S3 — the local diff view (`src/cli/diff.ts`, `uap diff`)
|
|
82
|
+
|
|
83
|
+
`gatherDiffFacts` is the only impure part: safe-git with `execFileSync`
|
|
84
|
+
**argument arrays** (no shell, no injection surface), a 5s timeout,
|
|
85
|
+
`core.fsmonitor=false -c core.untrackedCache=false` (defeat the stateful
|
|
86
|
+
caches that make status flaky on big repos), and **degrade-never-throw** —
|
|
87
|
+
a non-repo or an unusable git yields `gitError: "git unavailable or not a
|
|
88
|
+
repository"` and the renderer says exactly that, not a stack trace. One
|
|
89
|
+
numstat pass vs HEAD covers staged + unstaged; `git status --porcelain`
|
|
90
|
+
folds status codes onto the rows (worktree code preferred over index code
|
|
91
|
+
— the file's state on disk is what the operator sees) and collects
|
|
92
|
+
untracked paths; renames are normalized before matching because porcelain
|
|
93
|
+
prints `old -> new` while numstat prints `{old => new}`. `renderDiffView`
|
|
94
|
+
is pure and pinned by tests: rows sorted by additions, per-section caps
|
|
95
|
+
with honest "… and N more" counts, honest totals, binary files render `±`
|
|
96
|
+
(unknown — never `0`), ahead/behind against the merge-base, and an
|
|
97
|
+
`inspect:` footer with the follow-up commands.
|
|
98
|
+
|
|
99
|
+
### The bench report reader (`uap bench report`)
|
|
100
|
+
|
|
101
|
+
Reports accumulate under `benchmark-results/paired-*/`; the reader answers
|
|
102
|
+
"what did the latest run measure" (`uap bench report`, newest by
|
|
103
|
+
`report.json` mtime with a dirname tiebreak for bulk-copied trees) and
|
|
104
|
+
"did the delta hold" (`--compare`, the two newest runs). It unwraps the
|
|
105
|
+
harness's `{ ablation, report }` on-disk shape. Two rules carried over
|
|
106
|
+
from the reliability ladder, now enforced at the surface:
|
|
107
|
+
|
|
108
|
+
- **A delta is never printed without its CI, verdict, and McNemar
|
|
109
|
+
discordant count.**
|
|
110
|
+
- **An unusable run is NO MEASUREMENT.** `discrimination.usable === false`
|
|
111
|
+
prints the reason and the explicit warning line; the drift view warns
|
|
112
|
+
when *either* compared run could not measure. `--full` reprints the
|
|
113
|
+
stored report.md; `--json` emits the raw report.
|
|
114
|
+
|
|
115
|
+
## Consequences
|
|
116
|
+
|
|
117
|
+
- The operator's three daily questions each got a one-command answer whose
|
|
118
|
+
numbers all carry provenance — the same standard the ladder demands of
|
|
119
|
+
the agent.
|
|
120
|
+
- The S1a sample stream finally has a consumer, which retroactively
|
|
121
|
+
justifies its deliberate inclusion of synthetic zero-token rows in
|
|
122
|
+
`recentDurations` (duration/success data is real even when the
|
|
123
|
+
convergence-loop token totals are not visible at that seam).
|
|
124
|
+
- The estimator is pure and the renderers are pure, so the honesty rules
|
|
125
|
+
are pinned by fast unit tests rather than by CLI scraping.
|
|
126
|
+
- Test seams are documented as such: `--db` (`uap eta`), the `root`/`cwd`
|
|
127
|
+
options (`bench report` / `uap diff`) — they exist because vitest
|
|
128
|
+
workers cannot `process.chdir`, the lesson P4's sandbox-cli tests paid
|
|
129
|
+
for. No new env vars, no settings-registry entries, no schema changes.
|
|
130
|
+
- Known open surface (named, not hidden): the dashboard's Measured Pace
|
|
131
|
+
section is exercised through the estimator's tests rather than a
|
|
132
|
+
dashboard-level test — the dashboard command is environment-heavy and
|
|
133
|
+
the section is a thin formatting shell over the pinned estimator.
|
|
134
|
+
|
|
135
|
+
## Review round (all three reviewers' findings fixed in-branch)
|
|
136
|
+
|
|
137
|
+
The 3-parallel-reviewer pass (code-quality, security, architecture)
|
|
138
|
+
surfaced one design rule and several contract completions, all fixed
|
|
139
|
+
before merge:
|
|
140
|
+
|
|
141
|
+
- **Absence of data is not a claim of cleanliness.** `uap diff` records
|
|
142
|
+
which git reads succeeded (`numstatOk`/`statusOk` travel in the facts
|
|
143
|
+
and the JSON); a failed or timed-out read renders
|
|
144
|
+
`working tree: UNKNOWN`, never `clean`. Same rule in `bench report`:
|
|
145
|
+
one dangling symlink in `benchmark-results/` skips that entry, not the
|
|
146
|
+
whole listing. An unborn repo (fresh `git init`, no commits) is a repo —
|
|
147
|
+
probed with `--is-inside-work-tree`, not with a HEAD rev-parse, and its
|
|
148
|
+
branch reads via `symbolic-ref` (which works before the first commit).
|
|
149
|
+
- **`--full` degrades, never throws**: a run dir with `report.json` but
|
|
150
|
+
no `report.md` (interrupted write) prints the reason, falls back to the
|
|
151
|
+
delta summary, exits non-zero.
|
|
152
|
+
- **The read surfaces do not materialize what they read.** `estimatePace`
|
|
153
|
+
gained `requireExisting` — the reactor injection (per-prompt) and the
|
|
154
|
+
handsfree status line will not create an empty analytics store on a
|
|
155
|
+
checkout that has no history. The explicit `uap eta` command keeps the
|
|
156
|
+
store's established open semantics.
|
|
157
|
+
- **One open, not N**: key enumeration moved into the estimator layer as
|
|
158
|
+
fail-soft `listPaceKeys` (the `--all` path and the enumeration share a
|
|
159
|
+
single store open; the 50-key cap is named, not silent). The bench
|
|
160
|
+
reader split into `bench-report.ts` (one command per file, per the
|
|
161
|
+
repo's own pattern).
|
|
162
|
+
- **Terminal/prompt hygiene at one choke point** (`utils/terminal-safe.ts`):
|
|
163
|
+
every rendered foreign string (report fields, commit subjects, model
|
|
164
|
+
ids) is stripped of C0/C1/DEL control bytes; strings that become prompt
|
|
165
|
+
text are additionally flattened to one line and bounded — a hostile
|
|
166
|
+
model id cannot carry line-break structure into a block that presents
|
|
167
|
+
itself as measured data. The raw id still keys the store query.
|
|
168
|
+
- **Loud flags**: garbage `--limit`/`--half-life`/`--max` is refused with a
|
|
169
|
+
clean error (NaN never reaches the render math), and a `--base` that
|
|
170
|
+
would reach git as an *option* (refnames cannot start with `-`) is
|
|
171
|
+
refused rather than passed to argv. `--no-ext-diff` keeps repo-configured
|
|
172
|
+
diff drivers out of the reads.
|
|
173
|
+
- The prepass's four `execSync` template-literal findings in
|
|
174
|
+
`src/bin/cli.ts` are pre-existing debt outside this diff, verified
|
|
175
|
+
untouched by all three reviewers — they belong in a separate cleanup PR.
|
|
@@ -9,7 +9,7 @@ edits your main branch, writes plausible-but-wrong code, and cheerfully declares
|
|
|
9
9
|
"done" on something that never actually ran. UAP puts a station at each of those
|
|
10
10
|
break points so the work that comes off the line is trustworthy.
|
|
11
11
|
|
|
12
|
-
It ships as a single npm package (`@miller-tech/uap`,
|
|
12
|
+
It ships as a single npm package (`@miller-tech/uap`, v2.29.0) that installs the
|
|
13
13
|
`uap` CLI. This page gets it onto your machine and wired into your project.
|
|
14
14
|
|
|
15
15
|
## Prerequisites
|
|
@@ -132,6 +132,13 @@ Install UAP (`npm i -g universal-agent-protocol`) and every feature below activa
|
|
|
132
132
|
|
|
133
133
|
**Why it matters:** Fewer deploys, faster delivery, atomic rollouts.
|
|
134
134
|
|
|
135
|
+
### Operator Attention Queue (v2.29)
|
|
136
|
+
**What it does:** When a decision is genuinely the operator's (a placement parked with no auto path), the agent records an ASK — context plus the SAFE DEFAULT the asking surface vouches for — and keeps working. The AFK auto-decider adopts recorded safe defaults once the grace window lapses (logged, never silent) and escalates asks that have no safe default.
|
|
137
|
+
|
|
138
|
+
**When it kicks in:** At session start (the hook sweeps and shows what is waiting, plus what was auto-decided while you were away), on a 60s timer inside the dashboard, and whenever a gated surface parks on a decision.
|
|
139
|
+
|
|
140
|
+
**Why it matters:** The agent stops on any excuse — or worse, silently decides for you. Now an unattended run keeps moving on decisions that are safe to default, and every adoption is on the record (`uap attention list/decide`; `UAP_ATTENTION_OFF=1` silences the rail).
|
|
141
|
+
|
|
135
142
|
---
|
|
136
143
|
|
|
137
144
|
## Long-Task Autonomy (Hands-Free) ✨
|
|
@@ -270,6 +277,7 @@ long-running build is progressing — no guesswork.
|
|
|
270
277
|
| Pattern RAG | ✅ | Every prompt | Battle-tested implementations |
|
|
271
278
|
| Coordination | ✅ | Complex tasks | Parallel subagent execution |
|
|
272
279
|
| Deploy batching | ✅ | Multiple changes | Atomic, efficient deploys |
|
|
280
|
+
| Attention queue + AFK decider | ✅ | Operator decisions / session start / 60s dashboard timer | Asks instead of stalls; unattended adoptions logged |
|
|
273
281
|
| HALO | ✅ | Substantial prompts | Automatic task decomposition |
|
|
274
282
|
| Model presets | ✅ | Model selection | Optimal params per model |
|
|
275
283
|
|
package/docs/guides/PROXY.md
CHANGED
|
@@ -225,11 +225,12 @@ set is below.
|
|
|
225
225
|
|---|---|---|
|
|
226
226
|
| **Loop-breaker** | `PROXY_LOOP_BREAKER` (on) | Detects act/review cycles + no-progress streaks and releases `tool_choice` to `auto` so the model can actually terminate |
|
|
227
227
|
| **Stuck-break** | `PROXY_STUCK_BREAK` (on) | When the model self-reports "stuck in a loop" or hammers a rate-limited host, forces a terminal turn |
|
|
228
|
+
| **Error-loop** | `PROXY_ERROR_LOOP` (on), `PROXY_ERROR_LOOP_THRESHOLD` (3), `PROXY_ERROR_LOOP_HARD_LIMIT` (3) | The SAME tool-result error recurring despite varied edits: inject a STOP directive naming the failure — a KNOWN FIX when the class has one (the shared class table: hallucinated skill names, edit-anchor mismatches, identical old/new strings, truncated whole-file writes, policy blocks (where the remedy names the policy layer and never endorses error-named commands), HTTP 404s, missing DB credentials, missing pgcrypto), else the class-matched diagnose remedy. Directives ride the transcript tail as a user-role turn, carry an untrusted-tool-output caveat, and yield to recon-convergence — the 2026-09-24 opencode+qwen log shows 20 hard stops where three system-top re-read directives were ignored en route to each. After HARD_LIMIT ignored directives the session hard-fails (400); a fresh human user turn re-arms. Yields to lockstep when lockstep owns the shape |
|
|
228
229
|
| **Deferral-break** | `PROXY_DEFERRAL_BREAK` (on) | The inverse: a no-tool turn that *defers* work ("I need more exploration cycles") is forced to act on the next turn |
|
|
229
230
|
| **Recon-convergence** | `PROXY_RECON_CONVERGENCE_THRESHOLD` (40) | After N tool-using turns with **no** write/deliverable, inject "stop exploring, produce the deliverable"; escalates to stripping tools if it persists (0 disables) |
|
|
230
231
|
| **Hard-finalize** | `PROXY_HARD_FINALIZE_TURNS` (40) | Absolute turn cap before a forced finalize |
|
|
231
232
|
| **MANDATE-DELIVER** | `PROXY_MANDATE_DELIVER` (on) | On the delivery-enforcer block marker, pin `tool_choice` to `deliver` so the next turn *must* route through the gated path — makes routing binding for weak models |
|
|
232
|
-
| **Lockstep escalation** | `PROXY_LOCKSTEP_BREAK` (on), `PROXY_LOCKSTEP_PIVOT_AT` (2), `PROXY_LOCKSTEP_HARD_AT` (4) | The identical tool call failing with the identical error: inject a pivot directive at streak 2, a final warning one fire later, then hard-fail the session (400) at streak 4 instead of drifting (both directives always precede the stop, so it lands at `max(HARD_AT, PIVOT_AT+2)`). Each tier appends to the loop-incident ledger (`UAP_LOOP_INCIDENTS`, default `~/.config/uap/loop-incidents.jsonl`, ~3 records per episode, no rotation) for `uap loops` to inspect and hand to `uap ideate`/`uap deliver
|
|
233
|
+
| **Lockstep escalation** | `PROXY_LOCKSTEP_BREAK` (on), `PROXY_LOCKSTEP_PIVOT_AT` (2), `PROXY_LOCKSTEP_HARD_AT` (4) | The identical tool call failing with the identical error: inject a pivot directive at streak 2, a final warning one fire later, then hard-fail the session (400) at streak 4 instead of drifting (both directives always precede the stop, so it lands at `max(HARD_AT, PIVOT_AT+2)`). Directives ride the transcript tail as a user-role turn (the salient position; the 2026-10-07 ledger showed system-top directives ignored twice) and carry a KNOWN FIX line for recurring error classes (hallucinated skill names, invalid todo statuses, interrupted calls, edit-anchor mismatches, truncated writes, policy blocks, HTTP 404s, missing DB credentials, missing pgcrypto). Each tier appends to the loop-incident ledger (`UAP_LOOP_INCIDENTS`, default `~/.config/uap/loop-incidents.jsonl`, ~3 records per episode, no rotation) for `uap loops` to inspect and hand to `uap ideate`/`uap deliver`; `uap loops --steer` distills the recurring classes into session-start context — shell harnesses read it from `session-start.sh`, opencode from the plugin's `session.created` handler — so a fresh session never re-enters a logged loop. A fresh human user turn re-arms the guard. Signatures anchor on the decisive traceback line — never the content-free header or exit-code wrapper that unrelated failures share — error tokens glued into larger names (openobserve-error-scan) never count, and a Read payload with no `is_error` flag is content, not failure; the DOUBLING-DOWN pivot rides the same tail placement. Disarmed if `PROXY_ERROR_LOOP=off` or `PROXY_DOUBLING_BREAK=off` (their trackers feed it) |
|
|
233
234
|
|
|
234
235
|
### Context & compaction
|
|
235
236
|
|
package/docs/guides/SANDBOX.md
CHANGED
|
@@ -67,13 +67,98 @@ tool whose name starts with a prefix in `PROXY_SANDBOX_UNREACHABLE_PREFIXES` (de
|
|
|
67
67
|
back to `WebFetch` / local reads instead. (This is the "strip unreachable browser MCP tools
|
|
68
68
|
from sandboxed sessions" behavior.)
|
|
69
69
|
|
|
70
|
+
## Per-session grants (P4: recorded authority)
|
|
71
|
+
|
|
72
|
+
`UAP_SANDBOX_ALLOW` widens the boundary but nobody consented on the record. Bridge Phase 4
|
|
73
|
+
adds a **consent ledger**: every extra writable hole is a *grant* — scoped to a session,
|
|
74
|
+
bounded by a TTL, stated with a reason, recorded with who consented, and audit-logged on
|
|
75
|
+
use.
|
|
76
|
+
|
|
77
|
+
```bash
|
|
78
|
+
# consent to one more hole for this session (reason is REQUIRED)
|
|
79
|
+
uap sandbox grant /var/cache/ccache --session my-session --reason "shared compiler cache" --ttl 3600
|
|
80
|
+
uap sandbox grants [--session <id>] [--json] # inspect
|
|
81
|
+
uap sandbox revoke sbg-557c11d5 # take it back
|
|
82
|
+
# then run with the session that owns the grants
|
|
83
|
+
uap sandbox --session my-session -- claude --dangerously-skip-permissions
|
|
84
|
+
```
|
|
85
|
+
|
|
86
|
+
Rules worth knowing:
|
|
87
|
+
|
|
88
|
+
- The `--` separator before the command is REQUIRED: without it the sandbox's own flags
|
|
89
|
+
(`--session`, `--net`, `--ttl`, `--reason`, `--json`) are parsed as sandbox options, not
|
|
90
|
+
passed through. The grant verbs are intercepted only when there is no `--` —
|
|
91
|
+
`uap sandbox -- revoke …` runs a binary named `revoke`, not the ledger verb.
|
|
92
|
+
- **Consent comes from the operator, never from inside a sandbox.** `grant`/`revoke` are
|
|
93
|
+
refused when `UAP_SANDBOX_ACTIVE=1` (that flag is set by the sandbox on the untrusted
|
|
94
|
+
side); listing stays available. And the ledger + audit files are mounted READ-ONLY into
|
|
95
|
+
every sandbox namespace — a bind-mounted file cannot be unlinked, replaced, or written
|
|
96
|
+
inside it — so a sandboxed agent can neither plant grant rows nor forge audit lines.
|
|
97
|
+
- A grant is refused without `--session`, `--reason`, or a grantable target (an existing
|
|
98
|
+
absolute path; `/`, `/home`, and `$HOME` itself are refused — an over-broad grant IS a
|
|
99
|
+
sandbox escape, checked after symlink resolution, and re-resolved at plan time to close
|
|
100
|
+
the swap race).
|
|
101
|
+
- `--ttl` fails CLOSED on garbage: a non-numeric or negative value is refused (not
|
|
102
|
+
silently upgraded to the 8h default); `--ttl 0` means "expires immediately" (floored to
|
|
103
|
+
1s). Grants default to **8h** and hard-cap at **7d**; an identical open grant is renewed,
|
|
104
|
+
not duplicated; expired grants are invisible to the sandbox. Every persist is
|
|
105
|
+
re-verified against a concurrent writer before success is reported.
|
|
106
|
+
- The ledger lives at the MAIN checkout's `.uap/sandbox-grants.json` (one ledger per
|
|
107
|
+
checkout, shared by every surface — the ADR 0010 anchor). Every sandbox run appends to
|
|
108
|
+
`.uap/sandbox-audit.jsonl`: which grants were used, which network mode engaged, and any
|
|
109
|
+
legacy `UAP_SANDBOX_ALLOW` entries labeled `unrecordedAllow` — still honored, now named.
|
|
110
|
+
Audit lines carry a monotonic `seq` and the file is BOUNDED: past 12,000 lines it
|
|
111
|
+
compacts to the 10,000 newest event lines — plus the triggering event and one
|
|
112
|
+
`audit-compaction` marker recording the drop, so the file lands at 10,002.
|
|
113
|
+
- **Tamper evidence (the ADR 0011 trust anchor).** Every recorded grant/renew/revoke also
|
|
114
|
+
writes a 0600 anchor OUTSIDE the checkout
|
|
115
|
+
(`$XDG_STATE_HOME/uap/sandbox-anchors/<hash-of-root>.json`) recording the hash of the
|
|
116
|
+
sanitized ledger. On load, the ledger is re-hashed and compared: if something with the
|
|
117
|
+
checkout's write reach changed it outside a recorded consent operation, `uap sandbox
|
|
118
|
+
grants` stamps the listing INTEGRITY MISMATCH (rows shown but unverified), new grants
|
|
119
|
+
are REFUSED, and a sandbox session opens ZERO grant-derived holes — the kernel boundary
|
|
120
|
+
itself is unaffected. Revoking still works (removing authority is the safe direction)
|
|
121
|
+
and re-anchors the remaining rows as the new baseline, naming the prior mismatch
|
|
122
|
+
loudly. `UAP_SANDBOX_ANCHOR_DIR` pins the anchor dir (tests, odd layouts).
|
|
123
|
+
- `UAP_SANDBOX_GRANTS_OFF=1` resolves zero grants; `UAP_SANDBOX_GRANTS` pins the ledger
|
|
124
|
+
file (tests, odd layouts).
|
|
125
|
+
|
|
126
|
+
See [ADR 0011](../architecture/adr/0011-sandbox-grants-and-network-egress-boundary.md) for
|
|
127
|
+
the design and its trust model (the ledger is shared-write state, so it is sanitized at
|
|
128
|
+
ingest AND on load — a planted row must not become a kernel-writable hole).
|
|
129
|
+
|
|
130
|
+
## Network egress boundary (P4: `--net off`)
|
|
131
|
+
|
|
132
|
+
The P0 sandbox shares the host network (the agent must reach the proxy). For runs that
|
|
133
|
+
should have **no network at all** — untrusted code, evals, data-only work — `--net off`
|
|
134
|
+
adds `--unshare-net` with nothing brought up: no TCP, no DNS, not even loopback, at the
|
|
135
|
+
kernel. Any `connect()` fails; a harness that needs the model fails loudly with connection
|
|
136
|
+
refused, which is the boundary working.
|
|
137
|
+
|
|
138
|
+
```bash
|
|
139
|
+
uap sandbox --net off -- npm test # kernel-enforced zero egress
|
|
140
|
+
UAP_SANDBOX_NET=off uap sandbox -- ... # default the mode for every run
|
|
141
|
+
```
|
|
142
|
+
|
|
143
|
+
Every run's actual mode (`shared` / `off`) is audit-logged. There is deliberately **no
|
|
144
|
+
`--net proxy`** (netns cut with a single hole to the proxy): rootless Linux cannot express
|
|
145
|
+
it — AF_UNIX sockets are network-namespace-scoped, bwrap closes every inherited fd, and
|
|
146
|
+
the userspace network stacks have no outbound allowlist. A boundary that only works with
|
|
147
|
+
the right helper installed would be a suggestion, not enforcement. The full analysis and
|
|
148
|
+
the rejected alternatives live in
|
|
149
|
+
[ADR 0011](../architecture/adr/0011-sandbox-grants-and-network-egress-boundary.md); the
|
|
150
|
+
model-path choke point that DOES hold is the proxy's `PROXY_SOCKET` (0600 AF_UNIX socket,
|
|
151
|
+
plus `uap proxy connect` to re-serve it on local TCP for clients that cannot speak
|
|
152
|
+
AF_UNIX).
|
|
153
|
+
|
|
70
154
|
## Escape hatches (operator-only)
|
|
71
155
|
|
|
72
156
|
| Variable | Effect |
|
|
73
157
|
|---|---|
|
|
74
158
|
| `UAP_SANDBOX_OFF=1` | Run the command as-is, no sandbox (for debugging). |
|
|
75
|
-
| `UAP_SANDBOX_ALLOW=/path:/other` | Extra writable prefixes. |
|
|
159
|
+
| `UAP_SANDBOX_ALLOW=/path:/other` | Extra writable prefixes — UNRECORDED (audited as `unrecordedAllow`; prefer `uap sandbox grant`). |
|
|
76
160
|
| `UAP_SANDBOX_WORKDIR=/path` | Override the resolved workdir. |
|
|
161
|
+
| `UAP_SANDBOX_NET=off` | Default the network mode for every run. Strictly `shared` or `off` — any other value fails LOUD (exit 2), never a silent downgrade to shared. |
|
|
77
162
|
| `PROXY_SANDBOX_UNREACHABLE_PREFIXES` | Change which tool prefixes are stripped (proxy side). |
|
|
78
163
|
|
|
79
164
|
Treat all of these as trusted-launch-environment settings — they widen the boundary, and
|
|
@@ -81,9 +166,11 @@ the model should never be able to set them for itself.
|
|
|
81
166
|
|
|
82
167
|
## Limits (be honest about them)
|
|
83
168
|
|
|
84
|
-
- **Network is
|
|
85
|
-
|
|
86
|
-
|
|
169
|
+
- **Network is restricted only in `--net off` mode.** In the default `shared` mode a
|
|
170
|
+
sandboxed agent can still make outbound network calls; the sandbox's default is a
|
|
171
|
+
*filesystem* boundary. Model-path gating is the proxy's job
|
|
172
|
+
(`ANTHROPIC_PASSTHROUGH_MODELS=__local_only__` for the cloud path, and the
|
|
173
|
+
`PROXY_SOCKET` unix surface for per-user permission gating).
|
|
87
174
|
- **The writable holes are real writable areas.** Anything under the workdir, `/tmp`, or
|
|
88
175
|
`~/.cache`/`~/.claude`/`~/.npm` can be modified — including caches that persist across
|
|
89
176
|
runs.
|
|
@@ -0,0 +1,176 @@
|
|
|
1
|
+
# Model-performance proof options — how to demonstrate UAP's benefits with honest numbers
|
|
2
|
+
|
|
3
|
+
**Status:** Proposed (a menu of options, ordered by cost; each names what it
|
|
4
|
+
would prove, what it would NOT prove, and its price — no option starts
|
|
5
|
+
without an explicit go)
|
|
6
|
+
**Date:** 2026-10-12
|
|
7
|
+
**Builds on:** [PAIRED_FINDINGS](../benchmarks/PAIRED_FINDINGS.md) (the honest
|
|
8
|
+
baseline: what has and has not replicated), the
|
|
9
|
+
[reliability ladder](../performance/reliability-ladder.md) (the underpowered-status
|
|
10
|
+
rule), [harness-engineering uplift](harness-engineering-uplift-2026-07-31.md)
|
|
11
|
+
(the mechanism priors), and [K3 uplift validation](k3-uap-uplift-validation.md)
|
|
12
|
+
(the frontier run).
|
|
13
|
+
|
|
14
|
+
## The claim we are actually proving
|
|
15
|
+
|
|
16
|
+
The measured history constrains what any honest plan can promise. The paired
|
|
17
|
+
harness established:
|
|
18
|
+
|
|
19
|
+
- UAP's gate loop delivers a large accuracy lift **only against base agents
|
|
20
|
+
that do not self-verify** (+20pp vs a single-shot model); against a strong
|
|
21
|
+
self-verifying harness the pooled accuracy effect is a wash
|
|
22
|
+
(**Δ 0.0pp [−5.5, +6.9]** — the CI excludes +20pp) at a token cost.
|
|
23
|
+
- UAP never significantly *hurt* accuracy on any surface measured.
|
|
24
|
+
- The one tier where payoff dwarfed cost in the ladder is the hardest one
|
|
25
|
+
(brutal tier 26.1% → 56.8%, ⚠ underpowered).
|
|
26
|
+
|
|
27
|
+
So "prove benefits and improvements to model performance" decomposes into
|
|
28
|
+
**four distinct, separately-provable claims** — and each option below is
|
|
29
|
+
mapped to the claim(s) it can actually carry:
|
|
30
|
+
|
|
31
|
+
| # | Claim | Where it is strongest |
|
|
32
|
+
|---|---|---|
|
|
33
|
+
| C1 | The gate loop lifts weak/non-self-verifying models and hardest-tier tasks | Local paired A/B |
|
|
34
|
+
| C2 | Memory + compounding surfaces lift long-horizon, multi-session work | Unmeasured gap — no existing number |
|
|
35
|
+
| C3 | Operator surfaces (attention, pace) improve throughput/stopping, not accuracy | Unmeasured gap |
|
|
36
|
+
| C4 | Frontier-model uplift under a canonical harness | The K3 plan |
|
|
37
|
+
|
|
38
|
+
**House rule (applies to every option):** every reported delta carries its
|
|
39
|
+
discrimination status (powered / underpowered / descriptive) and a paired
|
|
40
|
+
design — same model, same tasks, same seeds, only the UAP layer toggled.
|
|
41
|
+
An underpowered delta is labeled "directional, underpowered" and never quoted
|
|
42
|
+
as a finding. Analysis follows the established format: per-task paired rates,
|
|
43
|
+
bootstrap 95% CI, McNemar on discordant pairs.
|
|
44
|
+
|
|
45
|
+
## The options, ordered by cost
|
|
46
|
+
|
|
47
|
+
### O1 — Aggregate the delivery-gate telemetry we already record (free)
|
|
48
|
+
|
|
49
|
+
**Claims:** C1 (descriptive support) · **Cost:** hours of analysis, no runs ·
|
|
50
|
+
**Prereqs:** none — the data exists.
|
|
51
|
+
|
|
52
|
+
Every `deliver` outcome already records best-%-of-gates per attempt per model
|
|
53
|
+
per technique (the `[deliver]` lessons). Option: aggregate this corpus into a
|
|
54
|
+
measured table — model × technique × tier → gate-pass rate and token cost —
|
|
55
|
+
and publish it as the first *descriptive* performance dataset.
|
|
56
|
+
|
|
57
|
+
- **Would prove:** where in the model/technique matrix the protocol's gates
|
|
58
|
+
bind (and where they are overhead) — at population scale, no new runs.
|
|
59
|
+
- **Would NOT prove:** causality (no controlled arm); it is descriptive
|
|
60
|
+
evidence that directs where to spend O2's controlled runs.
|
|
61
|
+
- **Discrimination status:** descriptive by construction; label it that way.
|
|
62
|
+
|
|
63
|
+
### O2 — Targeted local paired replication where the payoff was (free, GPU-local)
|
|
64
|
+
|
|
65
|
+
**Claims:** C1 · **Cost:** GPU-hours only ·
|
|
66
|
+
**Prereqs:** the local stack (`uap bench paired`, the swe-bench-pro-gen suite).
|
|
67
|
+
|
|
68
|
+
Re-run the paired harness on the current local model stack, UAP on/off, but
|
|
69
|
+
**stratified to the cells the ladder says matter**: the hardest tier (the
|
|
70
|
+
only tier with a measured 2.2× payoff, still underpowered) and a weak
|
|
71
|
+
single-shot arm (the +20pp regime, to check it still holds on today's models).
|
|
72
|
+
|
|
73
|
+
- **Would prove:** whether C1 survives replication on the current stack with
|
|
74
|
+
enough attempts to graduate from "underpowered, directional" to a powered
|
|
75
|
+
finding — the single most valuable upgrade to the ladder.
|
|
76
|
+
- **Would NOT prove:** anything about memory, attention, or frontier models.
|
|
77
|
+
- **Discrimination status:** starts underpowered by design; the run count is
|
|
78
|
+
chosen from a power calculation on the pilot variance (the existing paired
|
|
79
|
+
format's bootstrap CI sizes the needed attempts).
|
|
80
|
+
|
|
81
|
+
### O3 — The multi-session memory experiment (free; the real gap)
|
|
82
|
+
|
|
83
|
+
**Claims:** C2 · **Cost:** GPU-hours + a task-suite design ·
|
|
84
|
+
**Prereqs:** a k-session task sequence where session k depends on k−1.
|
|
85
|
+
|
|
86
|
+
The strongest *unmeasured* claim. The harness-engineering ablation gives the
|
|
87
|
+
prior (long-term memory alone: +5.6pp on their harness) and the paired
|
|
88
|
+
findings could not test it (their tasks are single-session). Design: N task
|
|
89
|
+
*sequences* of k sessions each, same model, two arms — UAP memory on vs
|
|
90
|
+
wiped-between-sessions — measuring per-session success, tokens, and the
|
|
91
|
+
session-over-session trend. The paired Δ *and the slope difference* are the
|
|
92
|
+
two numbers.
|
|
93
|
+
|
|
94
|
+
- **Would prove:** whether UAP's memory system (recall-first, compounding,
|
|
95
|
+
one store across agents) delivers the multi-session lift C2 claims — this
|
|
96
|
+
is the repo's most defensible original contribution, so it deserves the
|
|
97
|
+
repo's most careful experiment.
|
|
98
|
+
- **Would NOT prove:** single-session accuracy (O2's territory).
|
|
99
|
+
- **Discrimination status:** no prior number exists; pilot first, then power
|
|
100
|
+
up. Sequence design is the hard part and is the first deliverable.
|
|
101
|
+
|
|
102
|
+
### O4 — Pace-injection stopping behavior (free, unit-level)
|
|
103
|
+
|
|
104
|
+
**Claims:** C3 · **Cost:** hours · **Prereqs:** the `eta:pace` reactor block
|
|
105
|
+
and `model_analytics.db`.
|
|
106
|
+
|
|
107
|
+
The measured-pace surfaces inject the agent's own EWMA estimate. Test whether
|
|
108
|
+
the injection changes *stopping* behavior: iterations past green, wall-clock
|
|
109
|
+
to stop, over-iteration rate — paired on/off on the same task set. This is a
|
|
110
|
+
behavior metric (throughput and token waste), deliberately not an accuracy
|
|
111
|
+
claim.
|
|
112
|
+
|
|
113
|
+
- **Would prove:** whether the pace surface earns its place (faster correct
|
|
114
|
+
stops = tokens saved with no accuracy loss).
|
|
115
|
+
- **Would NOT prove:** accuracy improvement; a null result here is also
|
|
116
|
+
publishable (it would justify demoting the surface).
|
|
117
|
+
- **Discrimination status:** underpowered at pilot scale; directional unless
|
|
118
|
+
powered up.
|
|
119
|
+
|
|
120
|
+
### O5 — Attention/AFK unattended-throughput (free)
|
|
121
|
+
|
|
122
|
+
**Claims:** C3 · **Cost:** hours · **Prereqs:** the attention queue + a
|
|
123
|
+
task set with operator-blocking asks.
|
|
124
|
+
|
|
125
|
+
Measure blocked-on-operator time per session with the queue on vs off (the
|
|
126
|
+
ask-instead-of-stop rail plus the AFK auto-decider): wall-clock to
|
|
127
|
+
completion, number of hard stops, safe-default adoption correctness (each
|
|
128
|
+
adoption is logged, so correctness is auditable).
|
|
129
|
+
|
|
130
|
+
- **Would prove:** the throughput claim — less idle time, more unattended
|
|
131
|
+
completions — and that auto-adopted defaults were correct (the safety
|
|
132
|
+
claim).
|
|
133
|
+
- **Would NOT prove:** accuracy improvement; the correctness audit bounds the
|
|
134
|
+
risk the throughput number would otherwise hide.
|
|
135
|
+
- **Discrimination status:** descriptive unless task count is powered.
|
|
136
|
+
|
|
137
|
+
### O6 — Frontier replication under a canonical harness (paid)
|
|
138
|
+
|
|
139
|
+
**Claims:** C4 · **Cost:** ~$5 smoke → ~$40–90 pilot → ~$900–4,300 full ·
|
|
140
|
+
**Prereqs:** the [K3 plan](k3-uap-uplift-validation.md), verbatim.
|
|
141
|
+
|
|
142
|
+
The frontier option is already written, phased, and gated on an explicit go:
|
|
143
|
+
Terminal-Bench 2.1, Factory Droid as harness, Kimi K3 via BYOK, paired
|
|
144
|
+
arms (baseline / AGENTS.md protocol / full proxy enforcement). Nothing here
|
|
145
|
+
supersedes it; it is the only option with an external, publicly-anchored
|
|
146
|
+
comparator (K3's published 88.3, harness caveat attached).
|
|
147
|
+
|
|
148
|
+
- **Would prove:** whether the protocol treatment lifts a frontier model
|
|
149
|
+
under a harness Factory itself has published methodology for.
|
|
150
|
+
- **Would NOT prove:** attribution to any single UAP surface (it is the
|
|
151
|
+
whole-protocol treatment); the paired Δ is the claim, the absolute score
|
|
152
|
+
is the caveated secondary.
|
|
153
|
+
- **Discrimination status:** 89 tasks × 5 attempts is powered for the
|
|
154
|
+
published convention's effect sizes; report Δ with bootstrap CI + McNemar.
|
|
155
|
+
|
|
156
|
+
## Suggested order
|
|
157
|
+
|
|
158
|
+
1. **O1** (free, hours of analysis, no risk) → publishes the descriptive dataset and
|
|
159
|
+
targets O2's strata.
|
|
160
|
+
2. **O2 hardest-tier replication** (free, targeted) → upgrades the ladder's
|
|
161
|
+
strongest row from underpowered to powered, or retires it honestly.
|
|
162
|
+
3. **O4 + O5** (free, small) → the C3 throughput numbers, including the
|
|
163
|
+
demote-if-null honesty.
|
|
164
|
+
4. **O3** (the gap) → design review first (the sequence design is its own
|
|
165
|
+
deliverable), then pilot, then power up.
|
|
166
|
+
5. **O6** (paid) → only after O1–O3 sharpen what the frontier arms should
|
|
167
|
+
emphasize, and only on the explicit go the K3 plan already requires.
|
|
168
|
+
|
|
169
|
+
## What no option here claims
|
|
170
|
+
|
|
171
|
+
None of these options will show UAP lifting a strong self-verifying agent's
|
|
172
|
+
single-session accuracy — the pooled 0.0pp replication already bounds that
|
|
173
|
+
claim, and the honest headline is the *conditional*: the protocol pays where
|
|
174
|
+
verification, memory, and operator attention are otherwise missing. A plan
|
|
175
|
+
that promised more would be exactly the unpowered-delta marketing the paired
|
|
176
|
+
findings exist to prevent.
|
package/docs/reference/API.md
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
# UAP Programmatic API Reference
|
|
2
2
|
|
|
3
|
-
> Public API of the `@miller-tech/uap` package. Version
|
|
3
|
+
> Public API of the `@miller-tech/uap` package. Version v2.29.0.
|
|
4
4
|
|
|
5
5
|
> **🏭 Where this fits:** Cross-cutting — the machine parts behind the line.
|
|
6
6
|
> **What it delivers:** when the `uap` CLI isn't the right shape for your job,
|