@miller-tech/uap 1.136.2 → 1.137.2
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/dist/.tsbuildinfo +1 -1
- package/dist/benchmarks/paired/report.d.ts +13 -0
- package/dist/benchmarks/paired/report.d.ts.map +1 -1
- package/dist/benchmarks/paired/report.js +29 -3
- package/dist/benchmarks/paired/report.js.map +1 -1
- package/dist/benchmarks/paired/types.d.ts +59 -0
- package/dist/benchmarks/paired/types.d.ts.map +1 -1
- package/dist/benchmarks/paired/types.js +60 -0
- package/dist/benchmarks/paired/types.js.map +1 -1
- package/dist/bin/cli.js +30 -0
- package/dist/bin/cli.js.map +1 -1
- package/dist/cli/deliver.d.ts.map +1 -1
- package/dist/cli/deliver.js +111 -7
- package/dist/cli/deliver.js.map +1 -1
- package/dist/cli/react.d.ts.map +1 -1
- package/dist/cli/react.js +18 -0
- package/dist/cli/react.js.map +1 -1
- package/dist/cli/self-tuning.d.ts +33 -0
- package/dist/cli/self-tuning.d.ts.map +1 -0
- package/dist/cli/self-tuning.js +139 -0
- package/dist/cli/self-tuning.js.map +1 -0
- package/dist/cli/tune-adapter.d.ts +8 -0
- package/dist/cli/tune-adapter.d.ts.map +1 -0
- package/dist/cli/tune-adapter.js +26 -0
- package/dist/cli/tune-adapter.js.map +1 -0
- package/dist/config/settings-registry.d.ts.map +1 -1
- package/dist/config/settings-registry.js +10 -0
- package/dist/config/settings-registry.js.map +1 -1
- package/dist/coordination/adaptation-signal.d.ts +19 -0
- package/dist/coordination/adaptation-signal.d.ts.map +1 -0
- package/dist/coordination/adaptation-signal.js +36 -0
- package/dist/coordination/adaptation-signal.js.map +1 -0
- package/dist/delivery/agentic-executor.d.ts +16 -1
- package/dist/delivery/agentic-executor.d.ts.map +1 -1
- package/dist/delivery/agentic-executor.js +45 -4
- package/dist/delivery/agentic-executor.js.map +1 -1
- package/dist/delivery/decompose.d.ts +27 -0
- package/dist/delivery/decompose.d.ts.map +1 -1
- package/dist/delivery/decompose.js +31 -1
- package/dist/delivery/decompose.js.map +1 -1
- package/dist/delivery/repair-escalation.d.ts +49 -0
- package/dist/delivery/repair-escalation.d.ts.map +1 -0
- package/dist/delivery/repair-escalation.js +83 -0
- package/dist/delivery/repair-escalation.js.map +1 -0
- package/dist/delivery/verifier-ladder.d.ts +47 -0
- package/dist/delivery/verifier-ladder.d.ts.map +1 -1
- package/dist/delivery/verifier-ladder.js +87 -1
- package/dist/delivery/verifier-ladder.js.map +1 -1
- package/dist/self-harness/mods.d.ts +20 -1
- package/dist/self-harness/mods.d.ts.map +1 -1
- package/dist/self-harness/mods.js +41 -0
- package/dist/self-harness/mods.js.map +1 -1
- package/dist/self-harness/orchestrator.js +4 -0
- package/dist/self-harness/orchestrator.js.map +1 -1
- package/dist/self-harness/transfer.d.ts.map +1 -1
- package/dist/self-harness/transfer.js +2 -0
- package/dist/self-harness/transfer.js.map +1 -1
- package/dist/self-tuning/flag-writer.d.ts +48 -0
- package/dist/self-tuning/flag-writer.d.ts.map +1 -0
- package/dist/self-tuning/flag-writer.js +102 -0
- package/dist/self-tuning/flag-writer.js.map +1 -0
- package/dist/self-tuning/flags.d.ts +88 -0
- package/dist/self-tuning/flags.d.ts.map +1 -0
- package/dist/self-tuning/flags.js +189 -0
- package/dist/self-tuning/flags.js.map +1 -0
- package/dist/self-tuning/index.d.ts +25 -0
- package/dist/self-tuning/index.d.ts.map +1 -0
- package/dist/self-tuning/index.js +25 -0
- package/dist/self-tuning/index.js.map +1 -0
- package/dist/self-tuning/judge.d.ts +66 -0
- package/dist/self-tuning/judge.d.ts.map +1 -0
- package/dist/self-tuning/judge.js +145 -0
- package/dist/self-tuning/judge.js.map +1 -0
- package/dist/self-tuning/llm-tuner.d.ts +83 -0
- package/dist/self-tuning/llm-tuner.d.ts.map +1 -0
- package/dist/self-tuning/llm-tuner.js +191 -0
- package/dist/self-tuning/llm-tuner.js.map +1 -0
- package/dist/self-tuning/orchestrator.d.ts +90 -0
- package/dist/self-tuning/orchestrator.d.ts.map +1 -0
- package/dist/self-tuning/orchestrator.js +126 -0
- package/dist/self-tuning/orchestrator.js.map +1 -0
- package/dist/self-tuning/paired-validator.d.ts +50 -0
- package/dist/self-tuning/paired-validator.d.ts.map +1 -0
- package/dist/self-tuning/paired-validator.js +78 -0
- package/dist/self-tuning/paired-validator.js.map +1 -0
- package/dist/self-tuning/profiles/index.d.ts +11 -0
- package/dist/self-tuning/profiles/index.d.ts.map +1 -0
- package/dist/self-tuning/profiles/index.js +15 -0
- package/dist/self-tuning/profiles/index.js.map +1 -0
- package/dist/self-tuning/profiles/opus48.d.ts +11 -0
- package/dist/self-tuning/profiles/opus48.d.ts.map +1 -0
- package/dist/self-tuning/profiles/opus48.js +27 -0
- package/dist/self-tuning/profiles/opus48.js.map +1 -0
- package/dist/self-tuning/profiles/qwen36.d.ts +11 -0
- package/dist/self-tuning/profiles/qwen36.d.ts.map +1 -0
- package/dist/self-tuning/profiles/qwen36.js +33 -0
- package/dist/self-tuning/profiles/qwen36.js.map +1 -0
- package/dist/self-tuning/quality-scorer.d.ts +90 -0
- package/dist/self-tuning/quality-scorer.d.ts.map +1 -0
- package/dist/self-tuning/quality-scorer.js +180 -0
- package/dist/self-tuning/quality-scorer.js.map +1 -0
- package/dist/self-tuning/realtime-adaptor.d.ts +78 -0
- package/dist/self-tuning/realtime-adaptor.d.ts.map +1 -0
- package/dist/self-tuning/realtime-adaptor.js +127 -0
- package/dist/self-tuning/realtime-adaptor.js.map +1 -0
- package/dist/self-tuning/run.d.ts +61 -0
- package/dist/self-tuning/run.d.ts.map +1 -0
- package/dist/self-tuning/run.js +122 -0
- package/dist/self-tuning/run.js.map +1 -0
- package/dist/self-tuning/search-reducer.d.ts +76 -0
- package/dist/self-tuning/search-reducer.d.ts.map +1 -0
- package/dist/self-tuning/search-reducer.js +349 -0
- package/dist/self-tuning/search-reducer.js.map +1 -0
- package/dist/self-tuning/tuning-profile.d.ts +72 -0
- package/dist/self-tuning/tuning-profile.d.ts.map +1 -0
- package/dist/self-tuning/tuning-profile.js +134 -0
- package/dist/self-tuning/tuning-profile.js.map +1 -0
- package/dist/telemetry/session-telemetry.d.ts +25 -0
- package/dist/telemetry/session-telemetry.d.ts.map +1 -1
- package/dist/telemetry/session-telemetry.js +32 -0
- package/dist/telemetry/session-telemetry.js.map +1 -1
- package/dist/types/config.d.ts +13 -0
- package/dist/types/config.d.ts.map +1 -1
- package/dist/types/config.js +3 -0
- package/dist/types/config.js.map +1 -1
- package/docs/INDEX.md +7 -4
- package/docs/design/LLM_SELF_TUNING_ANALYSIS.md +617 -0
- package/docs/guides/AUTOMATIC_FEATURES.md +8 -0
- package/docs/guides/DROIDS_AND_SKILLS.md +1 -1
- package/docs/guides/SELF_TUNING.md +137 -0
- package/docs/reference/CLI.md +262 -1
- package/docs/reference/CONFIGURATION_REFERENCE.md +24 -0
- package/docs/reference/FEATURES.md +20 -5
- package/package.json +1 -1
- package/src/policies/enforcers/__pycache__/_common.cpython-312.pyc +0 -0
- package/tools/agents/scripts/__pycache__/toolcall_path_normalizer.cpython-312.pyc +0 -0
- package/tools/agents/scripts/anthropic_proxy.py +355 -29
- package/tools/agents/scripts/confidence_escalation.py +60 -4
- package/tools/agents/tests/test_confidence_escalation.py +77 -0
|
@@ -0,0 +1,137 @@
|
|
|
1
|
+
# LLM Self-Tuning
|
|
2
|
+
|
|
3
|
+
Raise a small model (e.g. qwen3.6) toward Opus-level output by tuning UAP's own
|
|
4
|
+
flag surface with a closed, benchmark-validated loop. Where **Self-Harness**
|
|
5
|
+
(`uap self-harness`) mines failure traces and rewrites proxy/scaffold knobs to
|
|
6
|
+
fix *correctness* bugs, **Self-Tuning** (`uap tune`) optimizes the first-class
|
|
7
|
+
settings surface — recipes, hands-free, memory, concurrency, proxy guardrails —
|
|
8
|
+
against a *quality* signal.
|
|
9
|
+
|
|
10
|
+
> Design: [`docs/design/LLM_SELF_TUNING_ANALYSIS.md`](../design/LLM_SELF_TUNING_ANALYSIS.md).
|
|
11
|
+
|
|
12
|
+
## The idea
|
|
13
|
+
|
|
14
|
+
A small model can't reliably judge its own output, and pass/fail on a task suite
|
|
15
|
+
doesn't tell you how *good* the output is. Self-tuning adds:
|
|
16
|
+
|
|
17
|
+
1. **A quality signal** (P0) — a multi-dimensional `QualityScore` (correctness,
|
|
18
|
+
quality, efficiency, tool-reliability, planning → composite 0-100), scored by
|
|
19
|
+
a stronger **judge model** when configured, or deterministically from the run
|
|
20
|
+
metrics otherwise.
|
|
21
|
+
2. **An LLM-guided tuner** (P1) — a stronger model proposes small, interpretable
|
|
22
|
+
flag changes given the quality history; when no judge is available a real
|
|
23
|
+
**Gaussian-process Bayesian optimizer** (ARD kernel over the mixed
|
|
24
|
+
numeric/enum/bool flag space, Expected-Improvement acquisition) picks the next
|
|
25
|
+
config instead.
|
|
26
|
+
3. **A closed loop** (P2) — every proposal is applied, validated by a paired A/B
|
|
27
|
+
benchmark (candidate config vs current, quality-scored), and accepted only on
|
|
28
|
+
a statistically-supported quality lift with no held-out regression.
|
|
29
|
+
4. **Model profiles** (P3) — the best-known config per model family, persisted
|
|
30
|
+
and reused; bundled starters for qwen3.6 and Opus ship in the box.
|
|
31
|
+
5. **Real-time adaptation** (P4) — per-session flag adjustments emitted from live
|
|
32
|
+
signals (tool-failure spikes, quality dips, context pressure, RECON loops)
|
|
33
|
+
over the one live channel the proxy honors mid-session.
|
|
34
|
+
|
|
35
|
+
## Quick start
|
|
36
|
+
|
|
37
|
+
```bash
|
|
38
|
+
# Dry-run a tuning loop over the real-gate suite with a local model (GP-only,
|
|
39
|
+
# no judge). Reports the best config it found without committing.
|
|
40
|
+
uap tune --model qwen36-a3b --adapter opencode
|
|
41
|
+
|
|
42
|
+
# Same, but let a stronger judge model both SCORE quality and PROPOSE changes.
|
|
43
|
+
uap tune --model qwen36-a3b --adapter opencode --judge opus-4.8
|
|
44
|
+
|
|
45
|
+
# Commit the tuned config to .uap.json / .uap/proxy.env and save the profile.
|
|
46
|
+
uap tune --model qwen36-a3b --adapter opencode --judge opus-4.8 --apply
|
|
47
|
+
|
|
48
|
+
# Offline smoke test (deterministic mock adapter — no live model needed):
|
|
49
|
+
uap tune --adapter mock --max-iterations 3 --json
|
|
50
|
+
```
|
|
51
|
+
|
|
52
|
+
`uap self-harness tune` is an alias of `uap tune`.
|
|
53
|
+
|
|
54
|
+
### Key flags
|
|
55
|
+
|
|
56
|
+
| Flag | Meaning |
|
|
57
|
+
|------|---------|
|
|
58
|
+
| `--model <id>` | Executor model family to tune (default `qwen36-a3b`). |
|
|
59
|
+
| `--adapter <name>` | `mock \| opencode \| claude \| mini \| raw \| deliver`. |
|
|
60
|
+
| `--judge <id>` | Judge/tuner model. Falls back to `recipes.judge.model`, then GP-only. |
|
|
61
|
+
| `--phase <name>` | Force one search phase: `coarse \| medium \| fine \| combinatorial`. |
|
|
62
|
+
| `--max-iterations <n>` | Tuning-loop budget (default 6). |
|
|
63
|
+
| `--apply` | Commit accepted configs + save the profile (default: dry-run). |
|
|
64
|
+
| `--json` | Machine-readable result. |
|
|
65
|
+
|
|
66
|
+
## What gets tuned
|
|
67
|
+
|
|
68
|
+
The tunable surface (`src/self-tuning/flags.ts`) is a curated subset of the
|
|
69
|
+
settings registry, each with a search domain and a **dependency** so the
|
|
70
|
+
optimizer never wastes trials on an inactive knob (e.g. `recipes.fusionN` only
|
|
71
|
+
matters when `recipes.recipe` is `fusion`/`auto`, which itself needs
|
|
72
|
+
`recipes.enabled`):
|
|
73
|
+
|
|
74
|
+
- **recipes**: `enabled`, `recipe`, `confidenceThreshold`, `fusionN`, `allowSelfJudge`
|
|
75
|
+
- **hands-free**: `enabled`, `intensity`, `UAP_HANDSFREE_STAGNATION_LIMIT`
|
|
76
|
+
- **concurrency**: `modelConcurrency.slots`, `modelConcurrency.adaptive`
|
|
77
|
+
- **memory**: `memory.shortTerm.maxEntries`, `memory.patternRag.enabled`
|
|
78
|
+
- **verification**: `delivery.runtimeVerify`
|
|
79
|
+
- **proxy guardrails**: `PROXY_RECON_CONVERGENCE_THRESHOLD`, `PROXY_LOOP_BREAKER`, `PROXY_STUCK_BREAK`
|
|
80
|
+
|
|
81
|
+
All writes go through the same validation as `uap config set`, so an
|
|
82
|
+
out-of-range proposal is clamped — it can never corrupt `.uap.json`. Rejected
|
|
83
|
+
trials are rolled back to the exact prior bytes.
|
|
84
|
+
|
|
85
|
+
## Profiles
|
|
86
|
+
|
|
87
|
+
Accepted configs advance a versioned `TuningProfile` under
|
|
88
|
+
`.uap/self-tuning/<model>.json`. Bundled starter profiles seed a fresh run and
|
|
89
|
+
act as cross-model transfer priors:
|
|
90
|
+
|
|
91
|
+
- **qwen3.6** — guardrail-heavy, judge-backed fusion, aggressive hands-free.
|
|
92
|
+
- **Opus 4.8** — the quality upper bound (most guardrails relaxed).
|
|
93
|
+
|
|
94
|
+
The runtime source of truth is the typed constants in
|
|
95
|
+
`src/self-tuning/profiles/*.ts`; the sibling `*.json` files are human-readable
|
|
96
|
+
reference artifacts.
|
|
97
|
+
|
|
98
|
+
## Real-time adaptation (auto-on)
|
|
99
|
+
|
|
100
|
+
The proxy freezes `PROXY_*` at startup and has no reload endpoint, so mid-session
|
|
101
|
+
tuning rides a per-session **adaptation signal** file — the same mechanism as the
|
|
102
|
+
recipe signal. It is **on by default** and conservative: a signal is emitted only
|
|
103
|
+
when a live signal actually breaches a threshold, so a nominal session behaves
|
|
104
|
+
exactly as before.
|
|
105
|
+
|
|
106
|
+
Live signals map to hot-reloadable adjustments:
|
|
107
|
+
|
|
108
|
+
| Signal | Adjustment |
|
|
109
|
+
|--------|-----------|
|
|
110
|
+
| tool-failure rate ↑ / turn-quality ↓ | escalate this turn to the judge (fusion) |
|
|
111
|
+
| context-window utilization ↑ | converge sooner (lower recon threshold) |
|
|
112
|
+
| RECON no-write streak ↑ | force synthesis / deliver now |
|
|
113
|
+
|
|
114
|
+
### Turning it off
|
|
115
|
+
|
|
116
|
+
The reactor emitter is the effective master switch — with it off, no signal is
|
|
117
|
+
written, so the proxy has nothing to honor. Any of these opts out:
|
|
118
|
+
|
|
119
|
+
```bash
|
|
120
|
+
uap config set realtimeAdapt.enabled false # persistent, project-level (recommended)
|
|
121
|
+
export UAP_REALTIME_ADAPT=0 # per-shell override (wins over config)
|
|
122
|
+
uap config set PROXY_REALTIME_ADAPT false # also make the proxy ignore signals
|
|
123
|
+
```
|
|
124
|
+
|
|
125
|
+
Precedence for the emitter: explicit call arg → `UAP_REALTIME_ADAPT` env →
|
|
126
|
+
`.uap.json realtimeAdapt.enabled` → default **on**.
|
|
127
|
+
|
|
128
|
+
## How it relates to Self-Harness
|
|
129
|
+
|
|
130
|
+
| | Self-Harness (`uap self-harness`) | Self-Tuning (`uap tune`) |
|
|
131
|
+
|---|---|---|
|
|
132
|
+
| Input | failure traces (HALO + proxy log) | quality-scored benchmark A/B |
|
|
133
|
+
| Optimizes | correctness (proxy/llama env, scaffold, middleware) | quality (first-class settings) |
|
|
134
|
+
| Proposer | heuristic + transfer | LLM tuner or GP-BO |
|
|
135
|
+
| Signal | pass/fail + McNemar | composite quality delta |
|
|
136
|
+
| Shared | the paired-bench validation pipeline, the Mod DSL (now incl. `ConfigMod`), and the versioned-profile pattern |
|
|
137
|
+
```
|
package/docs/reference/CLI.md
CHANGED
|
@@ -29,7 +29,7 @@ runs, keeping `--help` fast.
|
|
|
29
29
|
| Command | Purpose |
|
|
30
30
|
|---------|---------|
|
|
31
31
|
| [`init`](#init) | Initialize agent context in the current project |
|
|
32
|
-
| [`setup`](#setup) |
|
|
32
|
+
| [`setup`](#setup) | Guided setup wizard (arrow-key), or `--profile maximum\|minimal\|custom`; backs up instruction files, extracts custom content |
|
|
33
33
|
| [`analyze`](#analyze) | Analyze project structure and emit metadata |
|
|
34
34
|
| [`generate`](#generate) | Generate or update CLAUDE.md and related files |
|
|
35
35
|
| [`update`](#update) | Update CLAUDE.md, memory system, and related components |
|
|
@@ -41,6 +41,19 @@ runs, keeping `--help` fast.
|
|
|
41
41
|
| [`expert-route`](#expert-route) | Recommend an expert droid chain for a task |
|
|
42
42
|
| [`deliver`](#deliver) | Convergence loop: iterate a model against gates until delivery |
|
|
43
43
|
| [`harness`](#harness) | HALO harness optimization over execution traces |
|
|
44
|
+
| [`bench`](#bench) | Controlled paired (UAP-on vs UAP-off) benchmark with CIs |
|
|
45
|
+
| [`self-harness`](#self-harness) | Self-improving harness: mine failures → propose → validate → commit |
|
|
46
|
+
| [`tune`](#tune) | LLM self-tuning: raise a small model toward Opus by tuning UAP flags |
|
|
47
|
+
| [`verify`](#verify) | Runtime verification gate: prove changed code actually runs |
|
|
48
|
+
| [`config`](#config) | Inspect and change every UAP setting (`.uap.json` / `.uap/proxy.env`) |
|
|
49
|
+
| [`design`](#design) | DESIGN.md interrogation, lint, and the off-token UI gate |
|
|
50
|
+
| [`proxy`](#proxy) | Reference-counted, session-scoped local inference proxy lifecycle |
|
|
51
|
+
| [`orchestrator`](#orchestrator) | Toggle the long multi-turn deliver orchestrator |
|
|
52
|
+
| [`handsfree`](#handsfree) (`hf`) | Hands-free persistence: loop any model to a 100% ledger |
|
|
53
|
+
| [`challenge`](#challenge) | Open multi-agent challenge with a significance-gated leaderboard |
|
|
54
|
+
| [`sandbox`](#sandbox) | Run a command with a kernel-enforced writable-dir boundary |
|
|
55
|
+
| [`react`](#react) | Resolve dynamic experts/skills/patterns for a lifecycle event (JSON) |
|
|
56
|
+
| [`status`](#status) | Show HALO trace collection state |
|
|
44
57
|
| [`ideate`](#ideate) | Divergent ideation (open-collider) for hard problems |
|
|
45
58
|
| [`coord`](#coord) | Agent coordination and status |
|
|
46
59
|
| [`agent`](#agent) | Agent lifecycle, work coordination, and communication |
|
|
@@ -379,6 +392,254 @@ The trace file defaults to `$UAP_HALO_TRACE_PATH` or `.uap/halo/traces.jsonl`.
|
|
|
379
392
|
|
|
380
393
|
---
|
|
381
394
|
|
|
395
|
+
## `bench`
|
|
396
|
+
|
|
397
|
+
Controlled paired benchmark — hold the base model + agent constant and toggle
|
|
398
|
+
UAP on/off over the same real-gate suite and seeds, then report paired deltas
|
|
399
|
+
with confidence intervals, a McNemar gate-value 2×2, and a cost–accuracy Pareto.
|
|
400
|
+
|
|
401
|
+
| Subcommand | Key flags | Purpose |
|
|
402
|
+
|------------|-----------|---------|
|
|
403
|
+
| `paired` | `--suite`, `--adapter <mock\|opencode\|claude\|mini\|raw\|deliver>`, `--model`, `--epochs`, `--concurrency`, `--ablation`, `--lazy`, `--seed`, `--iterations`, `--rope-margin`, `--out`, `--json` | Run the UAP-on vs UAP-off A/B; writes `records.jsonl` + Markdown/JSON reports |
|
|
404
|
+
|
|
405
|
+
```bash
|
|
406
|
+
uap bench paired --adapter opencode --model qwen36-a3b --epochs 5
|
|
407
|
+
uap bench paired --ablation # leave-one-out per UAP component
|
|
408
|
+
```
|
|
409
|
+
|
|
410
|
+
See [Paired Harness](../benchmarks/PAIRED_HARNESS.md) for the methodology.
|
|
411
|
+
|
|
412
|
+
---
|
|
413
|
+
|
|
414
|
+
## `self-harness`
|
|
415
|
+
|
|
416
|
+
Self-improving harness: mine model-specific failures from traces, propose a
|
|
417
|
+
bounded, reversible modification (the "Mod" DSL), validate it with a real paired
|
|
418
|
+
bench, and — with `--apply` — commit it and snapshot a versioned profile.
|
|
419
|
+
|
|
420
|
+
| Subcommand | Key flags | Purpose |
|
|
421
|
+
|------------|-----------|---------|
|
|
422
|
+
| `analyze` | `--records`, `--env`, `--transfer`, `--json` | Mine weaknesses + propose candidate Mods (read-only) |
|
|
423
|
+
| `run` | `--records`, `--suite`, `--heldout`, `--adapter`, `--model`, `--epochs`, `--max-candidates`, `--apply`, `--json` | The full loop; `--apply` commits + snapshots |
|
|
424
|
+
| `transfer` | `--transfer`, `--json` | List the cross-model transfer store |
|
|
425
|
+
| `mine-prod` | `--traces`, `--unit`, `--since`, `--pending`, `--json` | Mine production traces → enqueue proposals (never applies) |
|
|
426
|
+
| `pending` / `prune` | `--pending`, `--transfer`, `--json` | Inspect / ablation-prune the queue + store |
|
|
427
|
+
| `tune` | (alias of `uap tune`) | The LLM/GP flag self-tuning loop |
|
|
428
|
+
|
|
429
|
+
```bash
|
|
430
|
+
uap self-harness run --suite benchmarks/suites/real-gate --adapter opencode
|
|
431
|
+
```
|
|
432
|
+
|
|
433
|
+
See [Self-Harness (design)](../design/SELF_HARNESS.md).
|
|
434
|
+
|
|
435
|
+
---
|
|
436
|
+
|
|
437
|
+
## `tune`
|
|
438
|
+
|
|
439
|
+
LLM self-tuning — raise a small model (e.g. qwen3.6) toward Opus-level output by
|
|
440
|
+
tuning UAP's own flag surface (recipes, hands-free, memory, concurrency, proxy
|
|
441
|
+
guardrails) with a benchmark-validated closed loop. An LLM proposes small flag
|
|
442
|
+
changes when a judge model is configured; otherwise a Gaussian-process Bayesian
|
|
443
|
+
optimizer picks the next config. `uap self-harness tune` is an alias.
|
|
444
|
+
|
|
445
|
+
| Flag | Purpose |
|
|
446
|
+
|------|---------|
|
|
447
|
+
| `--model <id>` | Executor model family to tune (default `qwen36-a3b`) |
|
|
448
|
+
| `--adapter <name>` | `mock \| opencode \| claude \| mini \| raw \| deliver` |
|
|
449
|
+
| `--judge <id>` | Judge/tuner model; else `recipes.judge.model`, else GP-only |
|
|
450
|
+
| `--phase <name>` | Force one search phase: `coarse \| medium \| fine \| combinatorial` |
|
|
451
|
+
| `--max-iterations <n>` | Tuning-loop budget (default 6) |
|
|
452
|
+
| `--epochs`, `--concurrency`, `--iterations`, `--seed` | Paired-bench + stats controls |
|
|
453
|
+
| `--apply` | Commit accepted configs + save the profile (default: dry-run) |
|
|
454
|
+
| `--json` | Machine-readable result |
|
|
455
|
+
|
|
456
|
+
```bash
|
|
457
|
+
uap tune --model qwen36-a3b --adapter opencode --judge opus-4.8 --apply
|
|
458
|
+
uap tune --adapter mock --max-iterations 3 --json # offline smoke test
|
|
459
|
+
```
|
|
460
|
+
|
|
461
|
+
See [LLM Self-Tuning](../guides/SELF_TUNING.md).
|
|
462
|
+
|
|
463
|
+
---
|
|
464
|
+
|
|
465
|
+
## `verify`
|
|
466
|
+
|
|
467
|
+
Run the project's completion gates — including the **runtime execution gate**
|
|
468
|
+
that actually runs the changed code — against the current files and report
|
|
469
|
+
pass/fail. This is what the Stop hook calls to block "done" on code that never
|
|
470
|
+
ran.
|
|
471
|
+
|
|
472
|
+
| Flag | Purpose |
|
|
473
|
+
|------|---------|
|
|
474
|
+
| `-d, --dir <path>` | Project directory (default: cwd) |
|
|
475
|
+
| `--strict` | Treat "no verifiable gates" as a failure (fail-closed) |
|
|
476
|
+
| `--runtime-only` | Run only the cheap runtime execution gate |
|
|
477
|
+
| `--full` | Also run the expensive integration / deploy-dev tiers |
|
|
478
|
+
| `--gates <ids>` | Comma-separated rung subset (e.g. `build,test,execution`) |
|
|
479
|
+
| `--acceptance <specfile>` | LLM acceptance gate: judge behavioral completeness vs a spec |
|
|
480
|
+
| `--no-visual` | Skip the headless visual gate |
|
|
481
|
+
| `-m, --model`, `--endpoint` | Model + endpoint for the acceptance gate |
|
|
482
|
+
| `--timeout <ms>`, `--json` | Per-rung timeout; JSON output |
|
|
483
|
+
|
|
484
|
+
```bash
|
|
485
|
+
uap verify --runtime-only # prove the artifact runs
|
|
486
|
+
uap verify --acceptance spec.md --strict
|
|
487
|
+
```
|
|
488
|
+
|
|
489
|
+
---
|
|
490
|
+
|
|
491
|
+
## `config`
|
|
492
|
+
|
|
493
|
+
Inspect and change every UAP setting from one place — the single source of truth
|
|
494
|
+
is `src/config/settings-registry.ts`, which also generates the
|
|
495
|
+
[Configuration Reference](CONFIGURATION_REFERENCE.md).
|
|
496
|
+
|
|
497
|
+
| Subcommand | Purpose |
|
|
498
|
+
|------------|---------|
|
|
499
|
+
| `list` | All settings + current values |
|
|
500
|
+
| `get <key>` / `explain <key>` | Read / learn one setting |
|
|
501
|
+
| `set <key> <value>` | Change it (writes `.uap.json`, `.uap/proxy.env`, or prints a shell export) |
|
|
502
|
+
| `doctor` | Flag risky / sub-optimal settings |
|
|
503
|
+
| `wizard` | Interactive expert configurator (also `uap setup --profile custom`) |
|
|
504
|
+
| `docs` | Regenerate `docs/reference/CONFIGURATION_REFERENCE.md` from the registry |
|
|
505
|
+
|
|
506
|
+
```bash
|
|
507
|
+
uap config set recipes.enabled true
|
|
508
|
+
uap config set realtimeAdapt.enabled false # opt out of real-time adaptation
|
|
509
|
+
uap config doctor
|
|
510
|
+
```
|
|
511
|
+
|
|
512
|
+
---
|
|
513
|
+
|
|
514
|
+
## `design`
|
|
515
|
+
|
|
516
|
+
DESIGN.md integration: interrogate an existing UI into a design brief, lint UI
|
|
517
|
+
work against it, and gate off-token colors / off-scale spacing.
|
|
518
|
+
|
|
519
|
+
| Flag | Purpose |
|
|
520
|
+
|------|---------|
|
|
521
|
+
| `-d, --project-dir <path>` | Project directory (default: cwd) |
|
|
522
|
+
| `-o, --out <path>` | Output path for `interrogate` (default `DESIGN.md`) |
|
|
523
|
+
| `--force` | Overwrite an existing DESIGN.md |
|
|
524
|
+
| `-f, --file <path>` | Target file (lint/check) or `"old,new"` (diff) |
|
|
525
|
+
| `--json` | Machine-readable output |
|
|
526
|
+
|
|
527
|
+
---
|
|
528
|
+
|
|
529
|
+
## `proxy`
|
|
530
|
+
|
|
531
|
+
Reference-counted, session-scoped lifecycle for the local inference proxy (the
|
|
532
|
+
Anthropic-compatible gateway in front of a local llama.cpp/Qwen). Hooks
|
|
533
|
+
`ensure`/`release` it per session so it starts on demand and stops when the last
|
|
534
|
+
session leaves — but it never kills a proxy that systemd manages or that other
|
|
535
|
+
sessions still use.
|
|
536
|
+
|
|
537
|
+
`uap proxy [ensure | release | status | start | stop | restart | enable | disable]`
|
|
538
|
+
|
|
539
|
+
| Flag | Purpose |
|
|
540
|
+
|------|---------|
|
|
541
|
+
| `--client <id>` | Client/session id (defaults to the session env or parent pid) |
|
|
542
|
+
| `--client-pid <n>` | Long-lived agent pid for liveness (hooks pass `$PPID`) |
|
|
543
|
+
| `--port <n>` | Proxy port (default 4000 / `$PROXY_PORT`) |
|
|
544
|
+
| `--if-enabled` | No-op unless `.uap.json` `proxy.autostart` is true (hook-safe) |
|
|
545
|
+
| `--quiet` / `--json` | Suppress output (hooks) / machine-readable status |
|
|
546
|
+
|
|
547
|
+
```bash
|
|
548
|
+
uap proxy status --json
|
|
549
|
+
uap proxy restart # e.g. after changing PROXY_* in .uap/proxy.env
|
|
550
|
+
```
|
|
551
|
+
|
|
552
|
+
The proxy binds `127.0.0.1` by default; see the proxy `PROXY_*` settings in the
|
|
553
|
+
[Configuration Reference](CONFIGURATION_REFERENCE.md) and [Local Models](../guides/LOCAL_MODELS.md).
|
|
554
|
+
|
|
555
|
+
---
|
|
556
|
+
|
|
557
|
+
## `orchestrator`
|
|
558
|
+
|
|
559
|
+
Toggle the long multi-turn deliver orchestrator (blackboard decomposition +
|
|
560
|
+
epic controller for big autonomous builds). Persists to `.uap.json`
|
|
561
|
+
(`deliver.orchestrate`).
|
|
562
|
+
|
|
563
|
+
`uap orchestrator [on | off | auto | status]` — `auto` (default) engages it only
|
|
564
|
+
for large multi-epic work. See [Orchestrator & Hands-Free](../guides/ORCHESTRATOR.md).
|
|
565
|
+
|
|
566
|
+
---
|
|
567
|
+
|
|
568
|
+
## `handsfree`
|
|
569
|
+
|
|
570
|
+
Hands-free persistence (alias `hf`): drive any model to keep working until the
|
|
571
|
+
multi-epic build **completion ledger** is 100% done instead of stopping early.
|
|
572
|
+
Auto-on.
|
|
573
|
+
|
|
574
|
+
`uap handsfree [status | on | off | init | complete <id> | fail <id> | remaining | stop-check]`
|
|
575
|
+
|
|
576
|
+
| Flag | Purpose |
|
|
577
|
+
|------|---------|
|
|
578
|
+
| `--mission <text>` | Mission text for `init` |
|
|
579
|
+
| `--items <json>` | JSON array of ledger items for `init` |
|
|
580
|
+
|
|
581
|
+
See [Orchestrator & Hands-Free](../guides/ORCHESTRATOR.md).
|
|
582
|
+
|
|
583
|
+
---
|
|
584
|
+
|
|
585
|
+
## `challenge`
|
|
586
|
+
|
|
587
|
+
Open multi-agent challenge: a shared goal, verified submissions, and a
|
|
588
|
+
significance-gated leaderboard (scores within a ROPE margin of the leader are
|
|
589
|
+
ties, not wins).
|
|
590
|
+
|
|
591
|
+
| Subcommand | Key flags | Purpose |
|
|
592
|
+
|------------|-----------|---------|
|
|
593
|
+
| `open` | `--metric <name>`, `--rope-margin <x>`, `--lower-is-better` | Open a challenge with a shared goal + scoring metric |
|
|
594
|
+
| `submit` / `leaderboard` / `status` | — | Submit a verified result / view the ranked board / show state |
|
|
595
|
+
|
|
596
|
+
```bash
|
|
597
|
+
uap challenge open --metric tps --rope-margin 4
|
|
598
|
+
```
|
|
599
|
+
|
|
600
|
+
---
|
|
601
|
+
|
|
602
|
+
## `sandbox`
|
|
603
|
+
|
|
604
|
+
Run a command with a kernel-enforced workdir boundary (bubblewrap): only the
|
|
605
|
+
current dir + scratch are writable, so writes outside fail at the kernel — the
|
|
606
|
+
boundary `--dangerously-skip-permissions` cannot bypass.
|
|
607
|
+
|
|
608
|
+
```bash
|
|
609
|
+
uap sandbox -- <command> [args...]
|
|
610
|
+
```
|
|
611
|
+
|
|
612
|
+
See [Sandbox](../guides/SANDBOX.md).
|
|
613
|
+
|
|
614
|
+
---
|
|
615
|
+
|
|
616
|
+
## `react`
|
|
617
|
+
|
|
618
|
+
Resolve the dynamic UAP capabilities (experts / skills / patterns) for a
|
|
619
|
+
lifecycle event and emit JSON — the engine behind the per-prompt Reactor. Hook
|
|
620
|
+
adapters pipe a JSON `ReactorContext` on stdin; the flags below are for manual
|
|
621
|
+
invocation.
|
|
622
|
+
|
|
623
|
+
| Flag | Purpose |
|
|
624
|
+
|------|---------|
|
|
625
|
+
| `--event <event>` | `user-prompt \| session-start \| pre-tool \| post-tool \| stop \| session-end` |
|
|
626
|
+
| `--prompt <text>` | Prompt text (when not piping a JSON payload) |
|
|
627
|
+
| `-f, --files <files...>` | Changed files (routing signal) |
|
|
628
|
+
| `--inject-threshold <n>` | Min confidence to inject (default 0.30) |
|
|
629
|
+
| `--auto-spawn-threshold <n>` | Min confidence to auto-spawn an expert (default 0.80) |
|
|
630
|
+
| `--max-inject-chars <n>` | Inject character budget (default 1200) |
|
|
631
|
+
|
|
632
|
+
See [Reactor (auto-apply)](../design/UAP_REACTOR.md).
|
|
633
|
+
|
|
634
|
+
---
|
|
635
|
+
|
|
636
|
+
## `status`
|
|
637
|
+
|
|
638
|
+
Show HALO trace collection state (enabled, path, span count). `--json` for
|
|
639
|
+
machine-readable output. (Equivalent to `uap harness status`.)
|
|
640
|
+
|
|
641
|
+
---
|
|
642
|
+
|
|
382
643
|
## `ideate`
|
|
383
644
|
|
|
384
645
|
Divergent ideation (open-collider): generate non-trivial ideas for hard
|
|
@@ -598,6 +598,30 @@ The serving recipe the proxy applies (mirror of `recipes.recipe`, consumed by th
|
|
|
598
598
|
|
|
599
599
|
**Recommendation:** Keep in sync with `recipes.recipe`; `uap setup`/`uap config` write both.
|
|
600
600
|
|
|
601
|
+
### `realtimeAdapt.enabled`
|
|
602
|
+
|
|
603
|
+
| | |
|
|
604
|
+
|---|---|
|
|
605
|
+
| **Where** | `.uap.json` |
|
|
606
|
+
| **Type** | boolean |
|
|
607
|
+
| **Default** | `true` |
|
|
608
|
+
|
|
609
|
+
Real-time flag adaptation (LLM Self-Tuning P4): the reactor emits per-session adjustments from live signals (tool-failure/quality/context/RECON) so the proxy can escalate or converge mid-session. This is the effective master switch — off means no signal is emitted.
|
|
610
|
+
|
|
611
|
+
**Recommendation:** Leave on. It is conservative (emits only when a live signal breaches a threshold) and is the effective master switch — disabling it turns the whole feature off regardless of the proxy side.
|
|
612
|
+
|
|
613
|
+
### `PROXY_REALTIME_ADAPT`
|
|
614
|
+
|
|
615
|
+
| | |
|
|
616
|
+
|---|---|
|
|
617
|
+
| **Where** | `.uap/proxy.env` |
|
|
618
|
+
| **Type** | boolean |
|
|
619
|
+
| **Default** | `true` |
|
|
620
|
+
|
|
621
|
+
Proxy side of real-time adaptation: whether the serving proxy honors a fresh adaptation signal per request. Auto-on; harmless when no emitter is running (no signal to honor).
|
|
622
|
+
|
|
623
|
+
**Recommendation:** Leave on. Set `false` only to make the proxy ignore adaptation signals even when the reactor emits them.
|
|
624
|
+
|
|
601
625
|
## Dashboard
|
|
602
626
|
|
|
603
627
|
<a id="dashboard"></a>The live analytics server and its mutation token.
|
|
@@ -231,7 +231,19 @@ Hierarchical MCP router that collapses 150+ MCP tools to 2
|
|
|
231
231
|
| Subsystem | Description |
|
|
232
232
|
|-----------|-------------|
|
|
233
233
|
| Observability (`observability/halo-exporter.ts`) | Emits agent/LLM/tool spans as OTLP/OpenInference JSONL for the HALO engine. Opt-in via `UAP_HALO_TRACE`; zero-overhead when off. |
|
|
234
|
-
| Telemetry (`telemetry/session-telemetry.ts`) | Session-level telemetry capture. |
|
|
234
|
+
| Telemetry (`telemetry/session-telemetry.ts`) | Session-level telemetry capture, incl. per-turn quality estimates for real-time tuning. |
|
|
235
|
+
|
|
236
|
+
## Self-Improvement (`src/self-harness/`, `src/self-tuning/`) — Feedback
|
|
237
|
+
|
|
238
|
+
> **Stage: Feedback.** The floor doesn't just log its mistakes — it rewrites its
|
|
239
|
+
> own harness to stop making them. Two closed loops, both validated by the same
|
|
240
|
+
> paired benchmark so a "fix" is never accepted on faith.
|
|
241
|
+
|
|
242
|
+
| Subsystem | Description |
|
|
243
|
+
|-----------|-------------|
|
|
244
|
+
| Self-Harness (`self-harness/`) | Autonomous mine → propose → validate → decide loop over a bounded, reversible **Mod DSL** (proxy/llama env knobs, scaffold text, middleware, and settings-registry `ConfigMod`s). Cross-model transfer store; online production-trace mining. Fixes **correctness** bugs. `uap self-harness`. |
|
|
245
|
+
| Self-Tuning (`self-tuning/`) | `uap tune` — raises a small model toward Opus by tuning UAP's flag surface against a multi-dimensional **quality** score. LLM-guided proposals with a real Gaussian-process Bayesian-optimizer fallback; atomic rollback-safe flag writes; per-model tuning profiles (bundled qwen3.6 / Opus). |
|
|
246
|
+
| Real-time adaptation (`self-tuning/realtime-adaptor.ts`) | Per-session flag adjustments from live signals (tool-failure / quality / context pressure / RECON loop) over the adaptation-signal channel the proxy honors mid-session. **Auto-on**; opt out with `realtimeAdapt.enabled:false`. |
|
|
235
247
|
|
|
236
248
|
## Browser & Benchmarks — QC / Verify
|
|
237
249
|
|
|
@@ -267,7 +279,10 @@ similarity, and system resource detection.
|
|
|
267
279
|
|
|
268
280
|
The `uap` CLI is the single door into the whole factory (top-level commands):
|
|
269
281
|
`init`, `setup`, `analyze`, `generate`, `memory`, `patterns`, `worktree`,
|
|
270
|
-
`sync`, `droids`, `expert-route`, `deliver`, `harness` (HALO),
|
|
271
|
-
`
|
|
272
|
-
`
|
|
273
|
-
`
|
|
282
|
+
`sync`, `droids`, `expert-route`, `deliver`, `verify`, `harness` (HALO),
|
|
283
|
+
`bench`, `self-harness`, `tune`, `config`, `design`, `proxy`, `orchestrator`,
|
|
284
|
+
`handsfree` (`hf`), `challenge`, `sandbox`, `react`, `ideate`, `coord`, `agent`,
|
|
285
|
+
`deploy`, `task`, `compliance`, `coordination`, `skill`, `update`,
|
|
286
|
+
`dashboard` (alias `dash`), `model`, `mcp-router`, `hooks`, `tool-calls`,
|
|
287
|
+
`rtk`, `mcp-setup`, `schema-diff`, `policy`, `uap-omp`. See
|
|
288
|
+
[CLI reference](CLI.md) for every command and flag.
|
package/package.json
CHANGED
|
Binary file
|