@miller-tech/uap 2.45.0 → 2.48.0

package/docs/plans/droid-client-performance-options.md

@@ -0,0 +1,252 @@
+# Droid client performance options — the local qwen3.8 UAP fleet
+
+**Status:** Proposed (monitoring findings + a ranked option menu; nothing starts
+without an explicit go, per the [proof-options house rule](model-performance-proof-options.md))
+**Date:** 2026-11-14
+**Monitored:** the live Droid qwen3.8 UAP clients (3 hooked repos, 4 droid
+processes; the pay2u security-audit run live since 16:01) against
+`~/.uap/telemetry/tool-calls.db` — 20,002 tool calls, 2026-10-11 → now.
+
+## What the monitoring shows
+
+**The failure mass is one class.** 20.6% of all tool calls fail (4,118), and
+**83% of the failures are the write/edit family**:
+
+| Tool | Calls | Failed | Fail rate |
+|---|--:|--:|--:|
+| edit_file | 3,677 | 1,557 | 42.3% |
+| edit_range | 588 | 233 | 39.6% |
+| write_file | 4,690 | 1,597 | 34.1% |
+| run_bash | 2,251 | 600 | 26.7% |
+| list_dir | 1,475 | 81 | 5.5% |
+| read_file | 7,321 | 50 | **0.7%** |
+
+Reads are essentially perfect. The model writes blind. Decomposing the
+write/edit failures:
+
+- **stub-refusal (1,097)** — UAP's gutting guard catches the model shrinking a
+  real file into a stub (or writing N declared functions with empty bodies).
+  Each refusal is a saved file but a wasted round-trip; the error text already
+  says "use edit_file", and models re-send anyway.
+- **no-op (758)** — writes/edits that produced no change (idempotent retries).
+- **edit-miss (642)** — `old_string not found`: editing from memory instead of
+  reading the file first.
+- **path-not-found (502)** — plausible-but-wrong guessed paths
+  (`apps/api/src/repositories/installments_repository.ts`-shaped).
+- edit-ambiguous (180), truncated-write (22).
+- run_bash's 600 = 447 deliver-sandbox refusals + 153 exit-1.
+
+**Era trend (the arm is the biggest single factor):**
+
+| Date | Era | Fail rate |
+|---|---|---:|
+| 11-02 | worst benchmark block | 46.5% |
+| 11-11 | **v31 UD-IQ4_XS + Q8_0 engram live** | **8.7%** |
+| 11-13 | v31/Q8_0 measured + swap round | 10.4% |
+| 11-14 (today) | **abliterated-iq2_xs serving** (user-switched) | 17.3% |
+
+The best measured live day is the v31 arm; today the client fleet is served by
+**iq2_xs — the fleet's floor tier (2.86 bpw)**. That alone roughly doubles the
+tool-call failure rate vs the v31 era.
+
+**The frontier routes are dark.** The client's 25 frontier-named entries
+(CC Opus 4.5/4.6, GPT-5.1/5.2, Gemini 3, AG variants) all point at
+`localhost:8317` — **no listener is bound**. The only frontier-tagged outcomes
+in the delivery corpus (opus-4.8) died at 0% of gates in ≤5 turns (early route
+death, not model quality). So "Opus 5.5 / Fable 5.2 levels" currently has **no
+valid measured baseline on this box** — the honest reading of the local gap is
+the failure table above: frontier-class clients read before editing (~0
+edit-miss), don't gut files (~0 stub-refusal), and don't guess paths (~0
+path-not-found). Closing the write/edit class is closing the gap.
+
+## The options, ranked by cost
+
+### O-A — Serve the strongest arm for client sessions (free, today)
+
+**Cost:** one panel switch · **Expected effect:** the 8.7%-era failure rate
+instead of today's 17.3% (the arm is the largest single measured factor).
+
+Droid client work runs on v31 (UD-IQ4_XS + the Q8_0 engram — the measured-best
+long-prompt decode) until the **UD-Q3_K_XL arm** (staged 2026-11-14, registry
+`strata-unsloth-ud-q3_k_xl`) is measured and A/B'd; iq2_xs stays for batch and
+benchmark traffic where its throughput per GiB matters. The Q3_K_XL A/B is
+already required before that arm can place (fail-closed `unmeasured`).
+
+### O-B — Restore (or drop) the frontier route, then measure the real gap (free)
+
+**Cost:** operator time · **Effect:** turns "frontier levels" from a vibe into
+a number.
+
+Either bring the `:8317` CC proxy back up or delete the 25 dead entries so the
+client menu stops lying. Then ONE paired run (the
+[paired discipline](../benchmarks/PAIRED_FINDINGS.md): same tasks, same
+seeds, UAP layer identical, only the model route toggled) — local fleet vs one
+frontier route — gives the actual per-task gap and the failure-table delta.
+Without this, every "get to Opus/Fable levels" claim is unmeasurable.
+
+### O-C — Kill the three big failure classes at the tool layer (days; the 83%)
+
+**Cost:** small, contained proxy/tool-router changes · **Effect:** directly
+targets 2,241 of the 2,687 tool failures (stub-refusal + no-op + edit-miss +
+path-not-found + edit-ambiguous).
+
+1. **edit-miss → auto-context**: on `old_string not found`, the error returns
+   the target's actual content around the nearest match (or line numbers +
+   first lines). The model currently guesses again; give it the truth. This is
+   the single highest-yield fix (642 failures, and each one saved is also a
+   saved retry turn).
+2. **path-not-found → fuzzy-match hint**: return the nearest existing path(s)
+   in the error (the 502 guessed paths are near-misses of real files).
+3. **stub-refusal → forced-read gate**: after a gutting refusal, the next
+   write_file to that path requires a fresh read of the target in the same
+   turn context — make the correct path (edit_file) cheaper than re-sending.
+   The refusal text alone demonstrably does not stop the 1,097 re-sends.
+4. **no-op surfacing**: flag "this write produced zero change" at the tool
+   layer (758 idempotent retries today pass silently).
+
+These are LOCKSTEP-class fixes (the v2.33 known-fix-table family): each one
+needs a paired before/after on the same benchmark suite, reported with the
+house discrimination status.
+
+### O-D — Proxy-side terse-prompt shim for the strata arms (days)
+
+**Cost:** small proxy change · **Effect:** the Sharp-template gains
+(terseness, 2.7× faster median upstream) approximated for the arms that
+cannot take a serve-time template.
+
+The strata engine has no serve-time template override (GGUF-embedded), so the
+Sharp template round skipped all 4 strata arms. What the proxy CAN do is
+inject the Sharp **system-prompt** text (the terseness directive) into the
+wire when the client's system prompt is absent/overridable — the template's
+mechanical wins (fewer filler tokens, earlier tool calls) without the
+GGUF re-embed. Weaker than the real template; cheap to A/B.
+
+### O-E — Aggregate the delivery-gate corpus (free; proof-plan O1)
+
+**Cost:** hours · **Effect:** the model × technique × tier gate table that
+directs every option above.
+
+The `[deliver]` lessons already record best-%-of-gates per model per task;
+aggregate them into the descriptive table the
+[proof plan](model-performance-proof-options.md) O1 specified. This is also
+what tells us whether the qwen35-a3b tier (67-83% best-gates) is client-viable
+or benchmark-only.
+
+## What NOT to do (the measured history says so)
+
+- **More gates on the Droid path.** The paired data: against a self-verifying
+  harness the gate loop is a wash (Δ 0.0pp [−5.5, +6.9]) at token cost. The
+  frontier gap is not a gating problem; it is the failure table.
+- **Chasing frontier models for the interactive client before O-B.** No valid
+  baseline exists; the two opus-tagged runs died at 0% in ≤5 turns.
+- **Full-pin any strata arm** (the 10-28 OOM doctrine) — throughput gains die
+  with the box.
+
+## Sequencing
+
+O-A today (panel switch, zero risk) → O-C.1 (edit-miss auto-context — the
+single best fix) in the next proxy round → O-B when the frontier route is
+restored → O-C.2/3/4 + O-E as the following PR → O-D last (weakest
+expected value per hour).
+
+---
+
+## O-B MEASURED — the frontier baseline via Droid + GLM-5.3 (2026-11-14)
+
+The frontier route landed differently than planned: not Claude Code through
+the `:8317` CLIProxyAPI OAuth (its refresh token is expired; re-login is
+user-interactive), but **Droid itself on `glm-5.3` (Max plan)** — the user
+switched the session default to glm-5.3 and directed the baseline there. This
+is the same client shape the monitored local baseline used (a Droid session
+doing real tool work), which makes it the honest pair.
+
+**Method (paired discipline): same tasks, same seeds, same driver, same
+gates — only the model route toggled.**
+
+- **Driver:** `droid exec --auto high -o json`, identical flags and 8-minute
+  per-task ceiling for both arms.
+- **Frontier arm:** `-m glm-5.3` (Droid's own provider auth — no local key
+  material involved).
+- **Local arm:** `-m qwen3.8-flash-next-unsloth-ud-iq4_xs-v31` through the
+  UAP proxy (`:4000`) → the fleet. During the run the fleet was serving
+  **abliterated-iq2_xs** (a peer's live deliver run owned the box, so no arm
+  switch was made) — the local arm is therefore labeled iq2_xs, the fleet's
+  floor tier, NOT v31.
+- **Suite:** 6 fresh-dir tasks exercising the monitored failure classes —
+  create+verify (t1), precise-edit of a seeded bug (t2), path near-miss
+  `.yaml` vs `.yml` (t3), add-to-existing without gutting (t4), multi-file
+  rename (t5), ambiguous-match edit-one-of-two (t6). 16 deterministic gates
+  evaluated identically for both arms.
+- **Anatomy:** from each run's droid session transcript (per-tool-call
+  results, same failure classes as the monitoring round, adapted to
+  Droid-native result text).
+
+### Results (2026-11-14, 19:34–20:22 AEDT)
+
+**Gate outcomes — TIED 13/13.** On small, well-specified, gate-verifiable
+tasks the floor-tier arm produces correct results too. The paired gap is
+**not outcome quality on this suite — it is cost**:
+
+| Arm | Gates | Exec completes | Tool calls | Turns (mean) | Wall time (total) |
+|---|---:|---:|---:|---:|---:|
+| **glm-5.3** (frontier, Max) | 13/13 | 6/6 | **59** | 7.3 | **4.7 min** |
+| **abliterated-iq2_xs** (local fleet, floor tier, via shimmed UAP proxy) | 13/13 | 1/6 (5 hit the 480s ceiling after finishing the file work) | **1,394** | 45 on the one completed task | **47 min** |
+
+The local arm needed **~24x the tool-call volume** and **~10x the wall
+time** for identical gate outcomes, and it burned its ceiling on 5 of 6
+tasks (the substantive file work was done when the ceiling killed the
+session — the gates scored the artifacts, which is why the tie is real).
+One completed local task (t6: a 3-line edit) took **45 turns**; the
+frontier took 4 for the same task. The local arm also spawned 20 subagent
+Task calls + 5 Skill calls where the frontier used none.
+
+**Failure anatomy (per Droid-native tool results, both arms):**
+
+| Tool | glm-5.3 (calls / fail / rate) | iq2_xs (calls / fail / rate) |
+|---|---|---|
+| Edit | 7 / **0** / 0.0% | 66 / **0** / 0.0% |
+| Create (write) | 7 / **0** / 0.0% | 21 / **0** / 0.0% |
+| Read | 19 / 2 / 10.5% | 270 / 5 / 1.9% |
+| Execute (bash) | 14 / 2 / 14.3% | 866 / 16 / 1.8% |
+| TaskOutput (bg polls) | — | 65 / 55 / 84.6% |
+| Overall | 59 / 4 / **6.8%** | 1,394 / 76 / 5.5% (1.6% excl. killed-bg-polls) |
+
+Classes: frontier = 2 bash-failures (both its own runs of the
+resolver-broken t5 fixture — reproduced identically on the pristine
+template, corrected in the gate), 1 path-not-found (the deliberate t3
+`.yaml`→`.yml` near-miss — recovered within the task), 1 other. Local = 16
+bash, 5 path-not-found, 55 killed-background-poll errors from the
+ceiling kills.
+
+### What the baseline actually establishes
+
+1. **The frontier "levels" number on this stack, for this suite, is: same
+   gates, 1/24th the tool activity, 1/10th the wall time, zero edit/write
+   misses.** On well-scoped tasks the floor arm eventually converges too —
+   frontier value here is throughput and turn economy, not correctness.
+2. **The blind-write failure class did not fire on this suite** — neither
+   arm missed an edit or gutted a file (all 73 edit/create calls across
+   both arms succeeded). These are small tasks whose files fit in one
+   read; the monitored corpus's 42%-edit-miss anatomy lives in long,
+   context-heavy sessions where the model edits from memory. The suite
+   bounds the claim: frontier ≠ better edit *accuracy* here; it is faster
+   at the same accuracy. The 20.6% live-corpus failure anatomy remains
+   the measure of the long-session problem — O-C's fixes target that
+   instrument, not this one.
+3. **The instrument caveat is load-bearing**: these anatomy numbers are
+   Droid-native tool results; the monitoring corpus's 20.6% is the UAP
+   tool layer. Same failure classes, different instruments — rates are not
+   comparable across them, only within this paired table.
+
+Raw artifacts: `benchmark-results/ob-paired-droid-exec-2026-11-14/`
+(report.json in the corpus schema + the full run suite), regenerated by
+the archived scripts in `artifacts/`. The O-E corpus aggregator picks the
+report up as a paired row on its next run.
+
+**Caveats (labeled, not hidden):** n=6 tasks per arm is directional, not
+powered — treat per-task deltas as single observations and the aggregate as
+descriptive. The local arm ran under concurrent fleet load (the live peer
+deliver run), which inflates its latency but not its gate outcomes. The
+frontier anatomy counts Droid-native tool results, not the UAP tool layer the
+monitoring round measured — the classes are aligned but the instruments
+differ; cross-comparison of rates between the two instruments is qualitative.

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@miller-tech/uap",
-  "version": "2.45.0",
+  "version": "2.48.0",
   "description": "Autonomous AI agent memory system with CLAUDE.md protocol enforcement",
   "type": "module",
   "main": "dist/index.js",

package/tools/agents/scripts/anthropic_proxy.py

@@ -6752,6 +6752,38 @@ def _extract_images(content) -> list[dict]:
     return images
 
 
+# O-D — the terse-prompt shim for arms that cannot take a serve-time template
+# (harness plan docs/plans/droid-client-performance-options.md). The strata
+# engine has no --chat-template override: its template is GGUF-embedded, so the
+# Sharp-template round (inference-panel PR #4) skipped every strata arm. What
+# the proxy CAN do is what the template's own terseness block does at render
+# time: prepend the terse directive to the system prompt. Upstream measured the
+# full template at 2.7x faster medians at the same solve rate; this shim is
+# the approximation, weaker by construction (no thinking-retention/tool-format
+# changes), and cheap to A/B by env.
+#   UAP_PROXY_TERSE_SYSTEM=0   — disable (the A/B off-arm)
+# The directive text is the Sharp template's terse block verbatim (its lead +
+# core), so the shim and the template say the same thing.
+_TERSE_SYSTEM_DIRECTIVE = (
+    "Answer directly and concisely. Give the answer with only what it needs to be "
+    "correct and usable.\n"
+    "Never: open with preamble or pleasantries; restate the question; add filler "
+    "transitions; hedge with niceties; or repeat a point you've already made.\n"
+    "Always: keep essential steps, caveats, uncertainties, and specifics — never "
+    "drop correctness or a needed warning for brevity. Keep the final answer lean. "
+    "Use the least structure that conveys it (plain prose when short; lists or code "
+    "only when they earn their place). If genuinely uncertain, say so and explain "
+    "why — never omit uncertainty for the sake of brevity.\n"
+    "If a user request is genuinely ambiguous, ask a sharp question, don't guess."
+)
+
+
+def _terse_system_enabled() -> bool:
+    """Default ON, opt-out by env — mirrors the Sharp template's own
+    default-on `terse` kwarg (opt out per request there, per machine here)."""
+    return os.environ.get("UAP_PROXY_TERSE_SYSTEM", "1") != "0"
+
+
 def anthropic_to_openai_messages(anthropic_body: dict) -> list[dict]:
     """Convert Anthropic message format to OpenAI message format.
 
@@ -6763,17 +6795,32 @@ def anthropic_to_openai_messages(anthropic_body: dict) -> list[dict]:
     """
     messages = []
 
-    # Anthropic has system as a top-level param
+    # Anthropic has system as a top-level param (string or content blocks).
+    # O-D: when the terse shim is on, the directive is PREPENDED to the
+    # caller's system text — matching the Sharp template's own behaviour of
+    # merging the terse block into the caller's system prompt. A caller's
+    # system prompt is never replaced, only led. When the client sent NO
+    # system prompt, NOTHING is injected: the proxy's standing doctrine
+    # (test_no_tools_does_not_inject_agentic_system_message) forbids
+    # inventing a system message the caller did not send, terse or otherwise.
     system = anthropic_body.get("system")
-    if system:
-        if isinstance(system, str):
-            messages.append({"role": "system", "content": system})
-        elif isinstance(system, list):
-            text = "\n".join(
-                b.get("text", "") for b in system if b.get("type") == "text"
+    system_text = ""
+    if isinstance(system, str):
+        system_text = system
+    elif isinstance(system, list):
+        system_text = "\n".join(
+            b.get("text", "") for b in system if b.get("type") == "text"
+        )
+    if system_text:
+        if _terse_system_enabled():
+            messages.append(
+                {
+                    "role": "system",
+                    "content": _TERSE_SYSTEM_DIRECTIVE + "\n\n" + system_text,
+                }
             )
-            if text:
-                messages.append({"role": "system", "content": text})
+        else:
+            messages.append({"role": "system", "content": system_text})
 
     for msg in anthropic_body.get("messages", []):
         role = msg["role"]