@miller-tech/uap 1.217.0 → 1.218.0

package/config/llama-profiles/qwen38-27b-dflash2.env

@@ -0,0 +1,211 @@
+# =====================================================================
+# Qwen 3.8 27B (dense, UD-IQ4_XS) @ 131072 ctx/slot / 2 slots, DFlash2
+# speculative decoding -- 2026-08-20. THIS IS THE DEFAULT PROFILE.
+# Service: uap-llama-server.service -> scripts/run-llama-server-continuity.sh
+# Install: cp to ~/.config/uap/llama-server.env
+#
+# Supersedes qwen38-27b-mtp.env (kept alongside as the fallback). Same model,
+# same context, same slot count -- ONLY the speculative-decoding config differs.
+#
+# MEASURED, not assumed (RTX 3090 24 GiB, sm_86, temp 0, 4 prompts,
+# 1032 tokens/arm, same binary across arms so spec config is the only variable):
+#
+#   arm                          aggregate tok/s   accept   tok/verify step
+#   draft-mtp   --parallel 1              59.3     93.6%          2.68
+#   DFlash2 n4  --parallel 1              68.7     67.4%          3.68   1.16x
+#   draft-mtp   --parallel 2              66.0     92.5%          2.67
+#   DFlash2 n4  --parallel 2              89.8     66.5%          3.66   1.36x
+#
+# The --parallel 2 row is two GENUINELY CONCURRENT requests, i.e. the
+# production shape. Note this CONTRADICTS the upstream PR thread, whose
+# reviewers report DFlash2 degrading at -np>=2; on this card the advantage
+# GROWS under concurrency. Re-measure before trusting it on other hardware.
+#
+# WHY n-max 4 AND NOT 7: n-max 7 was measured too and was SLOWER (69.8 vs
+# 76.7 tok/s in the first pass) despite accepting a longer block (4.86 vs
+# 3.87 tokens/step). Acceptance falls from 72% to 55% as the block lengthens,
+# so the extra drafting work outruns what it buys. The drafter card's
+# published acceptance length of 5.39 is real but is not the throughput
+# optimum here.
+#
+# VRAM -- THE REAL COST. Peak in this exact shape (262144 total ctx,
+# --parallel 2, both slots driven concurrently) is 23449 / 24576 MiB, i.e.
+# ~1.1 GiB free. The drafter's weights are the obvious cost, but the MARGINAL
+# cost vs draft-mtp is NOT measured: no paired probe of the mtp profile was ever
+# taken with the same harness, and docs/guides/QWEN36_LLAMACPP.md reports ~0.95
+# GiB free for draft-mtp at this same tier -- which cannot both be true and
+# leave DFlash2 roomier. Do not quote a delta until both are re-measured with
+# one harness. The 23449 figure is also a 1 s nvidia-smi sample over a ~10 s run
+# whose longest prefill was 540 tokens; it never saw a long-context prefill on
+# two slots, which is the actual peak event at 131072/slot.
+#
+# KNOWN DEFECT OF THIS BUILD -- SHUTDOWN HANGS. The PR-27342 binary can deadlock
+# in __futex_wait after printing "cleaning up before exit...", so SIGTERM does
+# not always complete. Under systemd this is contained: TimeoutStopSec=20 then
+# SIGKILL, and the cgroup releases the VRAM. Started OUTSIDE systemd it is not:
+# observed 2026-08-20 leaving an orphaned llama-server holding 21362 MiB with no
+# listening socket, which starves the real service on load. Always drive this
+# profile through `systemctl --user {start,stop,restart} uap-llama-server`, and
+# if a load fails with an allocation error check
+# `nvidia-smi --query-compute-apps=pid,used_memory --format=csv` for an orphan
+# first. This card has a history of GLOBAL kernel
+# OOM events selecting this service for its RSS (see the --cache-ram note
+# below), and the GPU side is now thinner too. If the server starts dying or
+# failing to load, switch back to qwen38-27b-mtp.env FIRST -- that is the
+# whole reason it is kept.
+#
+# ENGINE IS AN UNMERGED UPSTREAM PR. LLAMA_BIN below is built from
+# ggml-org/llama.cpp#27342 ("support DFlash2", commit 5ecbe1ac1) cherry-picked
+# onto f466cfa38 with zero conflicts. The patch is vendored at
+# config/llama-patches/dflash2-pr27342.patch so the build is reproducible if
+# that worktree is lost -- create a detached llama.cpp worktree at f466cfa38,
+# apply that patch to it, then configure with CUDA on, arch 86, native on,
+# shared libs on, Release, and build the llama-server target.
+# The merged draft-dflash v1 engine CANNOT substitute: a v2 checkpoint declares
+# the same general.architecture="dflash", so auto-detect picks draft-dflash and
+# then fails on the 23 conv/selector tensors and dflash.conv_*/selector_* keys
+# v1 has no loader for. Return to plain master once #27342 lands upstream.
+# =====================================================================
+LLAMA_BIN=/home/cogtek/llama.cpp/.worktrees/dflash2-trial/build-cuda/bin/llama-server
+LLAMA_MODEL=/home/cogtek/Downloads/Qwen3.8-27B-UD-IQ4_XS.gguf
+# CONFIRMED INCOMPATIBLE 2026-08-20: mmproj-BF16.gguf crashed the server on
+# load -- "mismatch between text model (n_embd = 5120) and mmproj
+# (n_embd = 2048)". It was built for a different, smaller model's vision
+# tower, not Qwen3.8-27B (n_embd 5120). Same applies to mmproj-F16.gguf
+# (same family of file, unverified for this model). Do not re-enable either
+# without a projector confirmed to report n_embd=5120 for its text side.
+# "none" is a real opt-out (added to the continuity script 2026-08-20); an
+# empty value is NOT equivalent -- it still falls through to auto-detect,
+# which is how this got picked up in the first place.
+LLAMA_MMPROJ=none
+
+LLAMA_HOST=0.0.0.0
+LLAMA_PORT=8080
+LLAMA_CTX_SIZE=262144
+LLAMA_PARALLEL=2
+# UAP RAILS -- KEEP IN SYNC WITH THESE TWO LINES (~/.config/uap/anthropic-proxy.env).
+# --ctx-size is the TOTAL across slots; /props reports the PER-SLOT figure.
+# 262144 / 2 slots = 131072 per slot, so the proxy rails must read:
+#   PROXY_CONTEXT_WINDOW=131072     (== per-slot ctx, NOT the total)
+#   PROXY_CONCURRENCY_LIMIT=2       (== LLAMA_PARALLEL, the slot count)
+# Over-stating either one makes the proxy admit work the server cannot hold and
+# turns a queue into context-overflow errors. Re-derive from /props after any
+# change here. LLAMA_PORT below must match the proxy's LLAMA_CPP_BASE port; the
+# proxy auto-discovers a live server if its pin is stale, but the pin should
+# still be correct so a cold boot does not depend on discovery.
+LLAMA_THREADS=16
+LLAMA_GPU_LAYERS=99
+
+LLAMA_CACHE_TYPE_K=q4_0
+LLAMA_CACHE_TYPE_V=q4_0
+
+LLAMA_BATCH_SIZE=512
+LLAMA_UBATCH_SIZE=512
+
+# DFlash2 speculative decoding. Unlike draft-mtp this needs a SEPARATE drafter
+# file -- z-lab/Qwen3.8-27B-DFlash2-GGUF Q4_K_M, trained for this exact target.
+# Q8_0 (2.0 GB) does NOT fit at this context tier; Q4_K_M (1.1 GB) does.
+#
+# The drafter carries its OWN 2048-token sliding window, so its KV cache is
+# ~80 MiB at f16 regardless of the target's 131072/slot -- which is why this
+# fits without cutting context or slots. Do not "fix" that by setting
+# LLAMA_DRAFT_CTX_SIZE; leaving it unset is what keeps the KV small.
+#
+# n-min and p-min are pinned to 0 DELIBERATELY. The measurement above was taken
+# with llama.cpp's defaults for a dflash drafter (n_min=0, p_min=0.00, per the
+# server's own startup line). The continuity script always passes both flags, so
+# leaving the draft-mtp values (1 / 0.75) here would gate the block-diffusion
+# drafter on a per-token probability it was never measured with, and the numbers
+# above would no longer describe what runs.
+LLAMA_ENABLE_SPEC_DECODING=true
+LLAMA_SPEC_TYPE=draft-dflash
+LLAMA_DRAFT_MODEL=/home/cogtek/Downloads/Qwen3.8-27B-DFlash2-Q4_K_M.gguf
+LLAMA_DRAFT_GPU_LAYERS=99
+LLAMA_DRAFT_MAX=4
+LLAMA_DRAFT_MIN=0
+LLAMA_DRAFT_P_MIN=0
+
+# Sampling requested for this profile (llama-server defaults are top-p 0.95,
+# top-k 40, temp 0.8 when unset -- see scripts/run-llama-server-continuity.sh).
+LLAMA_TEMP=0.7
+LLAMA_TOP_P=0.8
+LLAMA_TOP_K=20
+
+LLAMA_REPEAT_PENALTY=1.0
+# LLAMA_CACHE_REUSE stays unset here (carried over from the previous Qwen3.6
+# profile, where it never took effect). --cache-reuse (in-slot prompt PREFIX
+# reuse) is disabled at load time in server-context.cpp whenever
+# llama_kv_cache::get_can_shift() is false, which holds on three architectural
+# conditions, none of them a flag: arch==STEP35; n_pos_per_embd()>1 (M-RoPE);
+# or any layer with k_idx ("shifting would leave k_idx stale" -- the GDN
+# linear-attention layers). qwen35-arch models (3.6, 3.8) trip the latter.
+# The host-side prompt cache (--cache-ram) and --ctx-checkpoints are the
+# mechanisms that DO run here.
+LLAMA_SLOT_SAVE_PATH=/home/cogtek/.cache/inference/kv-slots
+LLAMA_LOG_FILE=/home/cogtek/llama.cpp/llama-server.log
+LLAMA_N_PREDICT=32768
+
+# CHAT TEMPLATE (2026-08-20): switched from qwen3.5-enhanced.jinja to
+# qwen-sharp.jinja -- peculiar-ragdoll/Qwen-Sharp-Chat-Templates
+# (template_version qwen3.8-froggeric-v22.1), vendored verbatim at
+# tools/agents/config/qwen-sharp.jinja. It targets Qwen 3.5/3.6/3.8 and is the
+# template for ALL qwen models here. Pinned artifact -- both are asserted in
+# test/chat-template-system-message.test.ts so a silent re-vendor fails CI:
+#   upstream revision  3dc34df52c63dd22ada21f96435e069deaa8d7da (2026-08-18)
+#   file sha256        d1f22a89eac3609dcfaa7b471b1f7d23bee2f084d275d26f4f8231d1d7908f4e
+#
+# WHAT IT FIXES: qwen3.5-enhanced.jinja hard-raised
+# 'No user query found in messages.' -- a server-side failure for the whole
+# request -- whenever no user turn survived its backwards scan, i.e. when every
+# user turn was a <tool_response> or when the transcript was system+assistant
+# only. Both shapes occur in normal agent loops. qwen-sharp renders them.
+#
+# WHAT STAYS COMPATIBLE (rendered and compared, not assumed):
+#   * the tool-call contract the model is told to EMIT is unchanged:
+#     <tool_call><function=NAME><parameter=K>V</parameter></function></tool_call>.
+#     anthropic_proxy.py's _HERMES_FUNCTION_RE / _TOOL_CALL_XML_RE parse it
+#     unchanged. The default is selected at render time by `_tool_format`, so
+#     the test asserts on RENDERED output -- the file contains a JSON emitter
+#     too, and a grep cannot tell which one is live.
+#   * the generation prompt matches the old template in BOTH enable_thinking
+#     states, so --reasoning auto --reasoning-format deepseek is unaffected.
+#   * a mid-conversation system message renders inline in both templates.
+#
+# WHAT ACTUALLY DIFFERS -- known and accepted:
+#   * HISTORY rendering of prior tool calls differs in whitespace only: with
+#     string-form `arguments` (the proxy's path -- it json.dumps them) the old
+#     template emitted "}\n</function>", the new one "}</function>"; and the
+#     separator between two tool calls in one assistant turn went "\n" ->
+#     "\n\n". Neither changes what the parsers accept.
+#   * the injected tool-calling instruction block was rewritten upstream: it now
+#     requires a <think> block before each tool call and forbids prose before
+#     the call. Behavioural, deliberate, not a wire change.
+#   * a system message containing an IMAGE now raises. Unreachable via the
+#     proxy, which flattens system to a text string before dispatch
+#     (anthropic_proxy.py ~4709). Direct-to-:8080 OpenAI clients can hit it.
+#   * unknown message roles no longer raise; they render as "[role]: content".
+#   * prior <think> blocks are retained across turns. Upstream sells this as a
+#     prompt-cache win; on the PROXY path it is a no-op, because the proxy
+#     already strips <think> from prior assistant turns on purpose
+#     (anthropic_proxy.py ~4725: "Stripping breaks the copy cycle"). On the
+#     direct path it re-introduces exactly that cycle -- pass
+#     chat_template_kwargs.preserve_thinking=false there if it shows up.
+#
+# chat_template_kwargs.reasoning_effort accepts
+# none|minimal|low|medium|high|xhigh|max. NOTE: 'none' forces thinking OFF,
+# overriding enable_thinking=true.
+# Do NOT switch this to "embedded" -- the model's own baked-in template still
+# carries the 'System message must be at the beginning.' raise.
+LLAMA_CHAT_TEMPLATE_FILE=/home/cogtek/dev/miller-tech/universal-agent-protocol/tools/agents/config/qwen-sharp.jinja
+
+# --cache-ram sizing inherited unchanged from the previous profile (measured
+# working set was for a different model, but the host-RAM cache mechanism
+# and its OOM history apply regardless of which model is loaded on the GPU
+# side). Re-measure with the eviction grep in qwen36-35b-a3b.env's history
+# if this model's prefill/eviction pattern differs materially.
+#
+# FLAG DRIFT: upstream renamed --checkpoint-every-n-tokens (old b8780
+# spelling) to --checkpoint-min-step. An unknown flag makes llama-server
+# exit 1, and systemd then burns its restart budget looping on it. Re-check
+# every flag here against `llama-server --help` whenever LLAMA_BIN moves.
+LLAMA_EXTRA_ARGS=--metrics --jinja --prio 2 --reasoning auto --reasoning-format deepseek --reasoning-budget 8000 --ctx-checkpoints 64 --checkpoint-min-step 2048 --cache-ram 16384

package/docs/guides/QWEN36_LLAMACPP.md

@@ -172,43 +172,102 @@ overriding `enable_thinking=true`.
 Do **not** switch a Qwen profile to `embedded` — the model's own baked-in template
 still carries the `'System message must be at the beginning.'` raise.
 
-## Speculative decoding: what is running, and DFlash2
-
-Running today: `--spec-type draft-mtp`, reading the MTP head baked into `--model`
-(no drafter file). At `--spec-draft-n-max 2` a verification step accepts at most 3
-tokens; measured draft acceptance is 60-91%.
-
-**DFlash2 exists, fits this card at the current context tier, and is not
-installed.** Measured 2026-08-20 on the RTX 3090, holding `--parallel 2` and
-131072 ctx/slot fixed:
-
-| | |
-|---|---|
-| Engine | [llama.cpp#27342](https://github.com/ggml-org/llama.cpp/pull/27342), **open, not merged**. One commit (`5ecbe1ac1`, ~676 LOC). Cherry-picks onto our build commit `f466cfa38` with **zero conflicts** — our tree is 29 commits ahead of the PR base — and builds clean for `sm_86`. |
-| Drafter | [z-lab/Qwen3.8-27B-DFlash2-GGUF](https://huggingface.co/z-lab/Qwen3.8-27B-DFlash2-GGUF), built for this exact target. Q4_K_M 1.1 GB / Q8_0 2.0 GB. Published acceptance length 5.39 / 5.13. |
-| Why v1 can't be used instead | The merged `draft-dflash` v1 engine cannot load a v2 checkpoint. Both declare `general.architecture = "dflash"`, so auto-detect selects `draft-dflash` and then fails: the v2 GGUF carries 23 conv/selector tensors and `dflash.conv_*` / `dflash.selector_*` metadata keys v1 has no loader for. A rebuild is mandatory; there is no config-only path. |
-
-VRAM, the constraint that decides it. The drafter is 5 layers with its **own 2048
-sliding window**, so its KV cache is ~80 MiB at f16 regardless of the target's
-131072 — the cost is essentially just weights:
-
-| | measured / computed |
-|---|---|
-| Free VRAM at 131072/slot, `--parallel 2` | ~1.9 GiB (`nvidia-smi`: 22607 / 24576 MiB) |
-| Q4_K_M drafter (weights + KV + buffers) | ~1.25 GiB → **fits**, ~0.7 GiB margin |
-| Q8_0 drafter | ~2.1 GiB → does not fit at this tier |
-
-So it does not require dropping context or slot count. The open question is
-throughput, not memory: the PR's own reviewers report degradation at `-np >= 2`
-— the setting this box runs — and the headline 5.x acceptance figures are
-single-stream on Blackwell / Apple M5, not `sm_86`. The honest test is a paired
-A/B at fixed `-np 2` and 131072/slot against `draft-mtp` as configured today.
-Until that measurement exists, `draft-mtp` stays.
-
-> Note: the VRAM table under *Measured: Qwen3.8-27B* below was recorded from
-> llama.cpp's own buffer accounting (~0.95 GiB free); the ~1.9 GiB here is
-> `nvidia-smi` free memory on the whole card. They measure different things —
-> use the smaller one for headroom decisions.
+## Speculative decoding: DFlash2 is the default
+
+`config/llama-profiles/qwen38-27b-dflash2.env` is the default profile as of
+2026-08-20. `qwen38-27b-mtp.env` is kept as the fallback — same model, same
+context, same slot count; only the speculative-decoding config differs.
+
+Measured on this card (RTX 3090, sm_86, temp 0, 4 prompts, 1032 tokens per arm,
+**same binary across arms** so the spec config is the only variable):
+
+| arm | aggregate tok/s | accept rate | tokens/verify step | |
+|---|---|---|---|---|
+| no speculation, `-np 1` | 41.3 | — | 1.00 | baseline |
+| `draft-mtp` n-max 2, `-np 1` | 59.3 | 93.6% | 2.68 | |
+| **DFlash2 n-max 4, `-np 1`** | **68.7** | 67.4% | 3.68 | **1.16×** |
+| `draft-mtp` n-max 2, `-np 2` | 66.0 | 92.5% | 2.67 | |
+| **DFlash2 n-max 4, `-np 2`** | **89.8** | 66.5% | 3.66 | **1.36×** |
+
+The `-np 2` rows are two *genuinely concurrent* requests — the production shape.
+Re-verified end-to-end through `run-llama-server-continuity.sh` with the shipped
+profile: **92.6 tok/s**, tool calls parsing, peak 23459 MiB.
+
+**Read `1.36×` as a blend, not a constant.** That run pushed 4 prompts through a
+2-slot pool, so the last prompt always finishes alone and the wall covers two
+different regimes. Decomposing it from the server's own `launch_slot_`/`release`
+timeline:
+
+| regime | `draft-mtp` | DFlash2 n-max 4 | ratio |
+|---|---|---|---|
+| both slots decoding | 76.4 tok/s | 130.3 tok/s | **1.71×** |
+| one request in flight on the 2-slot server | 57.9 tok/s | 60.4 tok/s | **1.04×** |
+| the measured 4-prompt burst (what `1.36×` reports) | 66.0 | 89.8 | 1.36× |
+
+So the gain is real but duty-cycle dependent: it is ~1.7× while both slots are
+busy and close to nothing when only one request is in flight. Add prompts to the
+burst and the headline drifts up toward 1.71×; remove them and it drifts down.
+The `-np 1` row above is a `--parallel 1` server, which this service never runs —
+for the single-request case on the real 2-slot server use the 1.04–1.2× range,
+not 1.16×.
+
+Two caveats on the numbers, both unclosed:
+
+* **Every measurement was taken at `temperature 0.0`**, while this profile serves
+  at `LLAMA_TEMP=0.7 / TOP_P=0.8 / TOP_K=20`. Greedy is the most favourable
+  setting for speculative decoding, and it flatters a 4-token block more than a
+  2-token one, so `1.36×` is an upper bound for the sampling that actually runs.
+* **Acceptance is strongly prompt-dependent for DFlash2 and flat for MTP.**
+  Across the six prompts ever run, DFlash2 ranged 0.40–0.95 and `draft-mtp` sat
+  in a 0.91–0.97 band. The two lowest-acceptance prompts (prose explanation) were
+  never run against `draft-mtp`, so the regime where DFlash2 could lose is
+  untested. `n-max 7 < n-max 4` is direct evidence that drafting cost dominates
+  once acceptance falls.
+
+Two results here are worth keeping, because both contradict what the upstream
+material predicts:
+
+* **DFlash2 does not degrade under concurrency on this card — it improves.** The
+  PR thread reports throughput collapsing at `-np >= 2`; here the advantage grows
+  from 1.16× to 1.36× going from one slot to two. Do not carry that assumption to
+  other hardware in either direction; measure it.
+* **`n-max 4` beats `n-max 7`** (76.7 vs 69.8 tok/s), even though n-max 7 accepts
+  a longer block (4.86 vs 3.87 tokens/step). Acceptance falls from 72% to 55% as
+  the block lengthens, so the extra drafting outruns what it buys. The drafter
+  card's published acceptance length of 5.39 is real but is not the throughput
+  optimum.
+
+### The cost, and when to roll back
+
+Peak VRAM in the production shape is **23459 / 24576 MiB — about 1.1 GiB free**.
+There is **no paired measurement of `draft-mtp` taken the same way**, so treat the
+drafter's marginal cost as un-measured rather than as the ~0.8 GiB the weights
+imply: the table above reports ~0.95 GiB free for `draft-mtp` at this same tier,
+which cannot both be true and leave DFlash2 roomier. Re-measure both profiles
+with one harness before relying on either figure. Note also that 23459 was
+sampled with `nvidia-smi` on a 1 s loop over a ~10 s run whose longest prefill
+was 540 tokens — it never exercised a long-context prefill on two slots, which is
+the real peak event at 131072/slot. The drafter's weights (Q4_K_M,
+1.1 GB) are essentially the whole difference: its KV cache stays around 80 MiB
+because it carries its own 2048-token sliding window rather than scaling with the
+target's 131072/slot. That is what lets this fit without cutting context or slots
+— and it is why `LLAMA_DRAFT_CTX_SIZE` must stay unset.
+
+This card has a history of *global* kernel OOM events selecting this service for
+its RSS. The GPU side is now thinner too. **If the server starts failing to load
+or dying, switch `~/.config/uap/llama-server.env` back to `qwen38-27b-mtp.env`
+before changing anything else** — that is what the fallback profile is for.
+
+### The engine is an unmerged PR
+
+[llama.cpp#27342](https://github.com/ggml-org/llama.cpp/pull/27342) is **open**.
+The build is a cherry-pick of `5ecbe1ac1` onto `f466cfa38` (zero conflicts),
+vendored as `config/llama-patches/dflash2-pr27342.patch` so it is reproducible if
+the build worktree is lost. The merged `draft-dflash` **v1** engine cannot
+substitute: a v2 checkpoint declares the same `general.architecture = "dflash"`,
+so auto-detect selects `draft-dflash` and then fails on the 23 conv/selector
+tensors and `dflash.conv_*` / `dflash.selector_*` metadata keys v1 has no loader
+for. Return to plain upstream master once the PR lands.
 
 ## Point UAP at it
 

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@miller-tech/uap",
-  "version": "1.217.0",
+  "version": "1.218.0",
   "description": "Autonomous AI agent memory system with CLAUDE.md protocol enforcement",
   "type": "module",
   "main": "dist/index.js",