@miller-tech/uap 1.217.0 → 1.219.0

package/config/llama-profiles/qwen38-27b-dflash2.env

@@ -0,0 +1,211 @@
+# =====================================================================
+# Qwen 3.8 27B (dense, UD-IQ4_XS) @ 131072 ctx/slot / 2 slots, DFlash2
+# speculative decoding -- 2026-08-20. THIS IS THE DEFAULT PROFILE.
+# Service: uap-llama-server.service -> scripts/run-llama-server-continuity.sh
+# Install: cp to ~/.config/uap/llama-server.env
+#
+# Supersedes qwen38-27b-mtp.env (kept alongside as the fallback). Same model,
+# same context, same slot count -- ONLY the speculative-decoding config differs.
+#
+# MEASURED, not assumed (RTX 3090 24 GiB, sm_86, temp 0, 4 prompts,
+# 1032 tokens/arm, same binary across arms so spec config is the only variable):
+#
+#   arm                          aggregate tok/s   accept   tok/verify step
+#   draft-mtp   --parallel 1              59.3     93.6%          2.68
+#   DFlash2 n4  --parallel 1              68.7     67.4%          3.68   1.16x
+#   draft-mtp   --parallel 2              66.0     92.5%          2.67
+#   DFlash2 n4  --parallel 2              89.8     66.5%          3.66   1.36x
+#
+# The --parallel 2 row is two GENUINELY CONCURRENT requests, i.e. the
+# production shape. Note this CONTRADICTS the upstream PR thread, whose
+# reviewers report DFlash2 degrading at -np>=2; on this card the advantage
+# GROWS under concurrency. Re-measure before trusting it on other hardware.
+#
+# WHY n-max 4 AND NOT 7: n-max 7 was measured too and was SLOWER (69.8 vs
+# 76.7 tok/s in the first pass) despite accepting a longer block (4.86 vs
+# 3.87 tokens/step). Acceptance falls from 72% to 55% as the block lengthens,
+# so the extra drafting work outruns what it buys. The drafter card's
+# published acceptance length of 5.39 is real but is not the throughput
+# optimum here.
+#
+# VRAM -- THE REAL COST. Peak in this exact shape (262144 total ctx,
+# --parallel 2, both slots driven concurrently) is 23449 / 24576 MiB, i.e.
+# ~1.1 GiB free. The drafter's weights are the obvious cost, but the MARGINAL
+# cost vs draft-mtp is NOT measured: no paired probe of the mtp profile was ever
+# taken with the same harness, and docs/guides/QWEN36_LLAMACPP.md reports ~0.95
+# GiB free for draft-mtp at this same tier -- which cannot both be true and
+# leave DFlash2 roomier. Do not quote a delta until both are re-measured with
+# one harness. The 23449 figure is also a 1 s nvidia-smi sample over a ~10 s run
+# whose longest prefill was 540 tokens; it never saw a long-context prefill on
+# two slots, which is the actual peak event at 131072/slot.
+#
+# KNOWN DEFECT OF THIS BUILD -- SHUTDOWN HANGS. The PR-27342 binary can deadlock
+# in __futex_wait after printing "cleaning up before exit...", so SIGTERM does
+# not always complete. Under systemd this is contained: TimeoutStopSec=20 then
+# SIGKILL, and the cgroup releases the VRAM. Started OUTSIDE systemd it is not:
+# observed 2026-08-20 leaving an orphaned llama-server holding 21362 MiB with no
+# listening socket, which starves the real service on load. Always drive this
+# profile through `systemctl --user {start,stop,restart} uap-llama-server`, and
+# if a load fails with an allocation error check
+# `nvidia-smi --query-compute-apps=pid,used_memory --format=csv` for an orphan
+# first. This card has a history of GLOBAL kernel
+# OOM events selecting this service for its RSS (see the --cache-ram note
+# below), and the GPU side is now thinner too. If the server starts dying or
+# failing to load, switch back to qwen38-27b-mtp.env FIRST -- that is the
+# whole reason it is kept.
+#
+# ENGINE IS AN UNMERGED UPSTREAM PR. LLAMA_BIN below is built from
+# ggml-org/llama.cpp#27342 ("support DFlash2", commit 5ecbe1ac1) cherry-picked
+# onto f466cfa38 with zero conflicts. The patch is vendored at
+# config/llama-patches/dflash2-pr27342.patch so the build is reproducible if
+# that worktree is lost -- create a detached llama.cpp worktree at f466cfa38,
+# apply that patch to it, then configure with CUDA on, arch 86, native on,
+# shared libs on, Release, and build the llama-server target.
+# The merged draft-dflash v1 engine CANNOT substitute: a v2 checkpoint declares
+# the same general.architecture="dflash", so auto-detect picks draft-dflash and
+# then fails on the 23 conv/selector tensors and dflash.conv_*/selector_* keys
+# v1 has no loader for. Return to plain master once #27342 lands upstream.
+# =====================================================================
+LLAMA_BIN=/home/cogtek/llama.cpp/.worktrees/dflash2-trial/build-cuda/bin/llama-server
+LLAMA_MODEL=/home/cogtek/Downloads/Qwen3.8-27B-UD-IQ4_XS.gguf
+# CONFIRMED INCOMPATIBLE 2026-08-20: mmproj-BF16.gguf crashed the server on
+# load -- "mismatch between text model (n_embd = 5120) and mmproj
+# (n_embd = 2048)". It was built for a different, smaller model's vision
+# tower, not Qwen3.8-27B (n_embd 5120). Same applies to mmproj-F16.gguf
+# (same family of file, unverified for this model). Do not re-enable either
+# without a projector confirmed to report n_embd=5120 for its text side.
+# "none" is a real opt-out (added to the continuity script 2026-08-20); an
+# empty value is NOT equivalent -- it still falls through to auto-detect,
+# which is how this got picked up in the first place.
+LLAMA_MMPROJ=none
+
+LLAMA_HOST=0.0.0.0
+LLAMA_PORT=8080
+LLAMA_CTX_SIZE=262144
+LLAMA_PARALLEL=2
+# UAP RAILS -- KEEP IN SYNC WITH THESE TWO LINES (~/.config/uap/anthropic-proxy.env).
+# --ctx-size is the TOTAL across slots; /props reports the PER-SLOT figure.
+# 262144 / 2 slots = 131072 per slot, so the proxy rails must read:
+#   PROXY_CONTEXT_WINDOW=131072     (== per-slot ctx, NOT the total)
+#   PROXY_CONCURRENCY_LIMIT=2       (== LLAMA_PARALLEL, the slot count)
+# Over-stating either one makes the proxy admit work the server cannot hold and
+# turns a queue into context-overflow errors. Re-derive from /props after any
+# change here. LLAMA_PORT below must match the proxy's LLAMA_CPP_BASE port; the
+# proxy auto-discovers a live server if its pin is stale, but the pin should
+# still be correct so a cold boot does not depend on discovery.
+LLAMA_THREADS=16
+LLAMA_GPU_LAYERS=99
+
+LLAMA_CACHE_TYPE_K=q4_0
+LLAMA_CACHE_TYPE_V=q4_0
+
+LLAMA_BATCH_SIZE=512
+LLAMA_UBATCH_SIZE=512
+
+# DFlash2 speculative decoding. Unlike draft-mtp this needs a SEPARATE drafter
+# file -- z-lab/Qwen3.8-27B-DFlash2-GGUF Q4_K_M, trained for this exact target.
+# Q8_0 (2.0 GB) does NOT fit at this context tier; Q4_K_M (1.1 GB) does.
+#
+# The drafter carries its OWN 2048-token sliding window, so its KV cache is
+# ~80 MiB at f16 regardless of the target's 131072/slot -- which is why this
+# fits without cutting context or slots. Do not "fix" that by setting
+# LLAMA_DRAFT_CTX_SIZE; leaving it unset is what keeps the KV small.
+#
+# n-min and p-min are pinned to 0 DELIBERATELY. The measurement above was taken
+# with llama.cpp's defaults for a dflash drafter (n_min=0, p_min=0.00, per the
+# server's own startup line). The continuity script always passes both flags, so
+# leaving the draft-mtp values (1 / 0.75) here would gate the block-diffusion
+# drafter on a per-token probability it was never measured with, and the numbers
+# above would no longer describe what runs.
+LLAMA_ENABLE_SPEC_DECODING=true
+LLAMA_SPEC_TYPE=draft-dflash
+LLAMA_DRAFT_MODEL=/home/cogtek/Downloads/Qwen3.8-27B-DFlash2-Q4_K_M.gguf
+LLAMA_DRAFT_GPU_LAYERS=99
+LLAMA_DRAFT_MAX=4
+LLAMA_DRAFT_MIN=0
+LLAMA_DRAFT_P_MIN=0
+
+# Sampling requested for this profile (llama-server defaults are top-p 0.95,
+# top-k 40, temp 0.8 when unset -- see scripts/run-llama-server-continuity.sh).
+LLAMA_TEMP=0.7
+LLAMA_TOP_P=0.8
+LLAMA_TOP_K=20
+
+LLAMA_REPEAT_PENALTY=1.0
+# LLAMA_CACHE_REUSE stays unset here (carried over from the previous Qwen3.6
+# profile, where it never took effect). --cache-reuse (in-slot prompt PREFIX
+# reuse) is disabled at load time in server-context.cpp whenever
+# llama_kv_cache::get_can_shift() is false, which holds on three architectural
+# conditions, none of them a flag: arch==STEP35; n_pos_per_embd()>1 (M-RoPE);
+# or any layer with k_idx ("shifting would leave k_idx stale" -- the GDN
+# linear-attention layers). qwen35-arch models (3.6, 3.8) trip the latter.
+# The host-side prompt cache (--cache-ram) and --ctx-checkpoints are the
+# mechanisms that DO run here.
+LLAMA_SLOT_SAVE_PATH=/home/cogtek/.cache/inference/kv-slots
+LLAMA_LOG_FILE=/home/cogtek/llama.cpp/llama-server.log
+LLAMA_N_PREDICT=32768
+
+# CHAT TEMPLATE (2026-08-20): switched from qwen3.5-enhanced.jinja to
+# qwen-sharp.jinja -- peculiar-ragdoll/Qwen-Sharp-Chat-Templates
+# (template_version qwen3.8-froggeric-v22.1), vendored verbatim at
+# tools/agents/config/qwen-sharp.jinja. It targets Qwen 3.5/3.6/3.8 and is the
+# template for ALL qwen models here. Pinned artifact -- both are asserted in
+# test/chat-template-system-message.test.ts so a silent re-vendor fails CI:
+#   upstream revision  3dc34df52c63dd22ada21f96435e069deaa8d7da (2026-08-18)
+#   file sha256        d1f22a89eac3609dcfaa7b471b1f7d23bee2f084d275d26f4f8231d1d7908f4e
+#
+# WHAT IT FIXES: qwen3.5-enhanced.jinja hard-raised
+# 'No user query found in messages.' -- a server-side failure for the whole
+# request -- whenever no user turn survived its backwards scan, i.e. when every
+# user turn was a <tool_response> or when the transcript was system+assistant
+# only. Both shapes occur in normal agent loops. qwen-sharp renders them.
+#
+# WHAT STAYS COMPATIBLE (rendered and compared, not assumed):
+#   * the tool-call contract the model is told to EMIT is unchanged:
+#     <tool_call><function=NAME><parameter=K>V</parameter></function></tool_call>.
+#     anthropic_proxy.py's _HERMES_FUNCTION_RE / _TOOL_CALL_XML_RE parse it
+#     unchanged. The default is selected at render time by `_tool_format`, so
+#     the test asserts on RENDERED output -- the file contains a JSON emitter
+#     too, and a grep cannot tell which one is live.
+#   * the generation prompt matches the old template in BOTH enable_thinking
+#     states, so --reasoning auto --reasoning-format deepseek is unaffected.
+#   * a mid-conversation system message renders inline in both templates.
+#
+# WHAT ACTUALLY DIFFERS -- known and accepted:
+#   * HISTORY rendering of prior tool calls differs in whitespace only: with
+#     string-form `arguments` (the proxy's path -- it json.dumps them) the old
+#     template emitted "}\n</function>", the new one "}</function>"; and the
+#     separator between two tool calls in one assistant turn went "\n" ->
+#     "\n\n". Neither changes what the parsers accept.
+#   * the injected tool-calling instruction block was rewritten upstream: it now
+#     requires a <think> block before each tool call and forbids prose before
+#     the call. Behavioural, deliberate, not a wire change.
+#   * a system message containing an IMAGE now raises. Unreachable via the
+#     proxy, which flattens system to a text string before dispatch
+#     (anthropic_proxy.py ~4709). Direct-to-:8080 OpenAI clients can hit it.
+#   * unknown message roles no longer raise; they render as "[role]: content".
+#   * prior <think> blocks are retained across turns. Upstream sells this as a
+#     prompt-cache win; on the PROXY path it is a no-op, because the proxy
+#     already strips <think> from prior assistant turns on purpose
+#     (anthropic_proxy.py ~4725: "Stripping breaks the copy cycle"). On the
+#     direct path it re-introduces exactly that cycle -- pass
+#     chat_template_kwargs.preserve_thinking=false there if it shows up.
+#
+# chat_template_kwargs.reasoning_effort accepts
+# none|minimal|low|medium|high|xhigh|max. NOTE: 'none' forces thinking OFF,
+# overriding enable_thinking=true.
+# Do NOT switch this to "embedded" -- the model's own baked-in template still
+# carries the 'System message must be at the beginning.' raise.
+LLAMA_CHAT_TEMPLATE_FILE=/home/cogtek/dev/miller-tech/universal-agent-protocol/tools/agents/config/qwen-sharp.jinja
+
+# --cache-ram sizing inherited unchanged from the previous profile (measured
+# working set was for a different model, but the host-RAM cache mechanism
+# and its OOM history apply regardless of which model is loaded on the GPU
+# side). Re-measure with the eviction grep in qwen36-35b-a3b.env's history
+# if this model's prefill/eviction pattern differs materially.
+#
+# FLAG DRIFT: upstream renamed --checkpoint-every-n-tokens (old b8780
+# spelling) to --checkpoint-min-step. An unknown flag makes llama-server
+# exit 1, and systemd then burns its restart budget looping on it. Re-check
+# every flag here against `llama-server --help` whenever LLAMA_BIN moves.
+LLAMA_EXTRA_ARGS=--metrics --jinja --prio 2 --reasoning auto --reasoning-format deepseek --reasoning-budget 8000 --ctx-checkpoints 64 --checkpoint-min-step 2048 --cache-ram 16384