@miller-tech/uap 1.52.0 → 1.53.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +15 -1
- package/dist/.tsbuildinfo +1 -1
- package/dist/delivery/applier.d.ts +29 -1
- package/dist/delivery/applier.d.ts.map +1 -1
- package/dist/delivery/applier.js +169 -2
- package/dist/delivery/applier.js.map +1 -1
- package/docs/INDEX.md +3 -1
- package/docs/architecture/OVERVIEW.md +23 -2
- package/docs/benchmarks/PAIRED_FINDINGS.md +147 -0
- package/docs/benchmarks/README.md +22 -4
- package/package.json +1 -1
- package/src/policies/enforcers/__pycache__/_common.cpython-312.pyc +0 -0
package/README.md
CHANGED
|
@@ -121,7 +121,19 @@ UAP installs hooks into your agent harness, then mediates every tool call throug
|
|
|
121
121
|
|
|
122
122
|
## Benchmarks
|
|
123
123
|
|
|
124
|
-
|
|
124
|
+
The honest, controlled result (paired A/B — same model, tasks, and seeds,
|
|
125
|
+
toggling only UAP, with confidence intervals): **UAP's accuracy lift depends on
|
|
126
|
+
whether the base agent already self-verifies.**
|
|
127
|
+
|
|
128
|
+
| Baseline | UAP accuracy lift | |
|
|
129
|
+
|---|---|---|
|
|
130
|
+
| Agentic harness (self-tests) | **~0pp** (CI spans 0) | overhead only — value is efficiency/coordination |
|
|
131
|
+
| Non-agentic single-shot model | **+20pp** (78%→98%, 95% CI [+8,+32], p=0.008) | gate loop repairs edge-case bugs |
|
|
132
|
+
|
|
133
|
+
Run it yourself: `uap bench paired --adapter raw --suite benchmarks/suites/real-gate-gated`.
|
|
134
|
+
Full analysis: **[docs/benchmarks/PAIRED_FINDINGS.md](docs/benchmarks/PAIRED_FINDINGS.md)**.
|
|
135
|
+
|
|
136
|
+
<details><summary>Earlier uncontrolled Terminal-Bench numbers (confounded — see TBench Investigation)</summary>
|
|
125
137
|
|
|
126
138
|
| Metric | Baseline | With UAP | Δ |
|
|
127
139
|
|---|---|---|---|
|
|
@@ -130,6 +142,8 @@ Terminal-Bench 2.0, 12 representative tasks, UAP-on vs. baseline:
|
|
|
130
142
|
| Errors per task | 1.17 | 0.42 | **−68%** |
|
|
131
143
|
| Wall-clock (total) | 618s | 266s | **−57%** |
|
|
132
144
|
|
|
145
|
+
</details>
|
|
146
|
+
|
|
133
147
|
Methodology, raw runs, and cost analysis: **[docs/benchmarks/](docs/benchmarks/)**.
|
|
134
148
|
|
|
135
149
|
---
|