@miller-tech/uap 1.172.1 → 1.172.6

package/config/llama-profiles/gemma4-26b-a4b-mtp.env

@@ -0,0 +1,145 @@
+# =====================================================================
+# Gemma 4 26B-A4B (MoE) @ 262144 ctx — 2026-07-28
+# (filename keeps "-mtp" for continuity; MTP is wired but measured OFF, below)
+# Service: uap-llama-server.service -> scripts/run-llama-server-continuity.sh
+# Install: cp to ~/.config/uap/llama-server.env.gemma4-26b-a4b-mtp
+#          ~/.config/uap/switch-profile.sh gemma4-26b-a4b-mtp
+#
+# WHY THIS PROFILE
+# The 26B A4B is a Mixture-of-Experts: it holds 26B parameters of knowledge
+# but activates only ~4B per token. That buys dense-26B-class quality at
+# A4B-class decode cost, which is what makes a 250k-context agent loop
+# practical on a single 24 GB card.
+#
+# MTP (multi-token prediction) was the original reason for this profile, but it
+# measured as a no-op on this MoE — see the speculative-decoding block below for
+# the A/B. Its VRAM now funds the full 262144 context instead.
+#
+# BINARY
+# Upstream master (PR #23398 merged 2026-06-07, build b9549) now carries Gemma 4
+# MTP itself, so this no longer needs the pr23398 side-build. The latest-master
+# worktree build has GEMMA4 + GEMMA4_ASSISTANT, the peg-gemma4 tool-call grammar,
+# and the newer spec types (dflash, eagle3) in one binary.
+#
+# WARNING: flag spellings drift between builds. Upstream renamed the checkpoint
+# spacing flag to --checkpoint-min-step; the OLD pinned mainline build
+# (/home/cogtek/llama.cpp/build/bin, b8780) still wants --checkpoint-every-n-tokens
+# and the Qwen profile still passes that. Do NOT point the Qwen profile at this
+# binary without fixing its EXTRA_ARGS first — the server exits 1 on an unknown
+# flag, and systemd will burn its restart budget looping on it.
+#
+# FIT (RTX 3090, 24 GB)
+#   main IQ4_XS            ~13.6 GB
+#   MTP draft head Q8_0    ~0.44 GB
+#   mmproj F16 (vision)    ~1.14 GB
+#   KV q8_0/q8_0 @262144   ~4-5 GB  (Gemma is KV-cheap: most layers are
+#                                    sliding-window; only the global layers
+#                                    carry full context)
+#   => ~20-21 GB, leaving headroom for the ub2048 compute buffers.
+# If this OOMs, step LLAMA_CTX_SIZE down (262144 -> 196608 -> 131072) before
+# touching quant or dropping the draft head.
+# =====================================================================
+LLAMA_BIN=/home/cogtek/llama.cpp/.worktrees/latest-master/build/bin/llama-server
+LLAMA_MODEL=/home/cogtek/Downloads/gemma-4-26B-A4B-it-UD-IQ4_XS.gguf
+LLAMA_DRAFT_MODEL=/home/cogtek/Downloads/mtp-gemma-4-26B-A4B-it-Q8_0.gguf
+
+# Vision projector MUST be pinned. The launch script's auto-discovery falls
+# back to a bare `mmproj*F16.gguf` glob in the model dir, which would match
+# the Qwen projector sitting in the same directory and load a mismatched
+# vision tower. Pin the Gemma-matched one explicitly.
+LLAMA_MMPROJ=/home/cogtek/Downloads/mmproj-gemma-4-26B-A4B-F16.gguf
+
+LLAMA_HOST=0.0.0.0
+LLAMA_PORT=8080
+# Full native context (gemma4.context_length=262144), funded by dropping MTP.
+#
+# Sizing is measured, not guessed. From this server's own state-save lines,
+# KV costs 331.969 MiB for 17071 tokens = ~0.0194 MiB/token, so the extra
+# 131072 tokens cost ~2.5 GB — against the 2.0 GB that MTP gave back, i.e.
+# roughly a wash against the 21.1 GB the MTP build occupied.
+#
+# History worth keeping: 262144 @ --parallel 2 with ub2048 DID OOM earlier.
+# The mmproj is allocated LAST, so that failure surfaces as a vision-tower
+# cudaMalloc abort inside clip_model_loader rather than an obvious KV overflow
+# — do not read that error as "the projector is broken".
+#
+# If this OOMs, step to 196608 before touching quant or dropping vision.
+#
+# CONTEXT BUDGET — why parallel is 2, at zero VRAM cost
+# --ctx-size is the TOTAL across slots; each slot gets ctx_size / parallel, so
+# moving 1 -> 2 repartitions the SAME KV rather than allocating more. Delivery
+# tuning holds a session at <130k (PROXY_CONTEXT_WINDOW=130000), so a single
+# 262144 slot could never use more than half its own KV.
+#
+#   parallel 1 -> 1 x 262144 : 131072 of KV unreachable, 1 concurrent session
+#   parallel 2 -> 2 x 131072 : every slot at the target, 2 sessions  <-- optimal
+#   parallel 3 -> 3 x  87381 : slots fall BELOW the 130k target
+#
+# 3x131072 would need ctx 393216 ~= 7.6 GB of KV at the measured 0.0194
+# MiB/token, landing near 23.8/24.5 GB — too little margin, don't.
+LLAMA_CTX_SIZE=262144
+LLAMA_PARALLEL=2
+LLAMA_THREADS=16
+LLAMA_GPU_LAYERS=99
+
+# Gemma degrades under aggressive KV quant and is KV-cheap anyway (sliding
+# window on most layers) — q8_0/q8_0 is the right trade here, NOT q4_0.
+LLAMA_CACHE_TYPE_K=q8_0
+LLAMA_CACHE_TYPE_V=q8_0
+
+# Agent workloads are prefill-bound (big tool-result payloads), so ubatch is
+# worth VRAM — but ub2048 compute buffers are what tipped this card into OOM
+# alongside the vision tower. 1024 keeps most of the prefill win.
+LLAMA_BATCH_SIZE=1024
+LLAMA_UBATCH_SIZE=1024
+
+# --- MTP speculative decoding: OFF, and that is the measured choice ------
+# A/B on this exact box, 1200-token generations, 2 samples per arm:
+#   MTP on : 105.9 / 101.8 t/s   21071 MiB
+#   MTP off: 103.8 / 104.7 t/s   19047 MiB
+# MTP is worth nothing on this MoE and costs 2.0 GB. That matches the PR
+# author's own report (">2x on dense gemma4, no speed-up on the MoE") — the
+# A4B only activates ~4B params per token, so the trunk decode is already
+# cheap enough that verifying a draft costs about what generating did.
+#
+# The 2.0 GB is NOT just the 440 MB draft head: the MTP path allocates its own
+# f16 draft KV context (cache_k=f16, cache_v=f16, ctx_dft=yes) alongside it.
+# Spending that on context instead is what funds ctx 262144 below.
+#
+# Flip ENABLE_SPEC_DECODING back to true to re-test (e.g. on a dense Gemma 4,
+# where it genuinely is >2x); the draft path and its tuned params are kept
+# intact below so it is a one-word change.
+LLAMA_ENABLE_SPEC_DECODING=false
+LLAMA_SPEC_TYPE=draft-mtp
+LLAMA_DRAFT_MAX=4
+LLAMA_DRAFT_MIN=1
+LLAMA_DRAFT_P_MIN=0.7
+LLAMA_DRAFT_GPU_LAYERS=99
+
+# Gemma's own guidance: no repeat penalty.
+LLAMA_REPEAT_PENALTY=1.0
+LLAMA_CACHE_REUSE=256
+LLAMA_SLOT_SAVE_PATH=/home/cogtek/.cache/inference/kv-slots
+LLAMA_LOG_FILE=/home/cogtek/llama.cpp/llama-server.log
+LLAMA_N_PREDICT=16384
+
+# Gemma 4 speaks its own <|turn>/<|tool_call> DSL, NOT ChatML. Use the
+# model's embedded template — the qwen3.5-enhanced.jinja used by the Qwen
+# profile emits a format Gemma was never trained on and silently destroys
+# tool calling.
+LLAMA_CHAT_TEMPLATE_FILE=embedded
+
+# EXTRA_ARGS append last, so --temp here overrides the script's --temp 0.3.
+# Gemma sampling defaults: temp 1.0 / top-k 64 / top-p 0.95 / min-p 0.0.
+# --jinja is REQUIRED: it turns on server-side rendering + parsing of Gemma's
+# tool-call DSL into OpenAI-shaped tool_calls, which is what the proxy and
+# opencode consume. NOTE: no --reasoning-format deepseek here — that is a
+# Qwen <think> convention and does not apply to Gemma.
+#
+# FLAG COMPATIBILITY: these are the pr23398 build's spellings, which differ
+# from mainline 8780's. In particular the checkpoint-spacing flag here is
+# --checkpoint-min-step; mainline's --checkpoint-every-n-tokens does NOT exist
+# on this binary and makes the server exit 1 at startup. Do not copy EXTRA_ARGS
+# across from the Qwen profile (which runs on mainline) without re-checking
+# each flag against `llama-server --help` for THIS binary.
+LLAMA_EXTRA_ARGS=--metrics --jinja --mlock --prio 2 --temp 1.0 --top-k 64 --top-p 0.95 --min-p 0.0 --ctx-checkpoints 32 --checkpoint-min-step 2048 --cache-ram 8192

package/config/llama-profiles/qwen36-35b-a3b.env

@@ -0,0 +1,69 @@
+# =====================================================================
+# Qwen 3.6 35B-A3B (MoE, IQ4_XS) @ 260000 ctx / 2 slots — 2026-07-28
+# Service: uap-llama-server.service -> scripts/run-llama-server-continuity.sh
+# Install: cp to ~/.config/uap/llama-server.env.qwen36-35b-a3b
+#
+# CONTEXT BUDGET — the reason parallel is 2 and not 1 or 3
+# llama.cpp treats --ctx-size as the TOTAL across slots; each slot gets
+# ctx_size / parallel. Delivery tuning holds a session at <130k to stay in the
+# model's competent range (PROXY_CONTEXT_WINDOW=130000), so any KV beyond 130k
+# in a single slot is unreachable — it can never be filled.
+#
+#   parallel 1 -> 1 x 260000  : 130000 of KV permanently idle
+#   parallel 2 -> 2 x 130000  : exact fit, 2 concurrent sessions  <-- optimal
+#   parallel 3 -> 3 x  86666  : each slot now BELOW the 130k target
+#
+# So 2 is not a compromise, it is the only division that both fills the KV and
+# keeps every slot at the target window. Raising ctx to 390000 for 3x130000 was
+# tried previously and OOMs on this card.
+#
+# Keep PROXY_CONCURRENCY_LIMIT equal to the slot count. If the proxy admits more
+# concurrent contexts than there are slots, the extra one evicts a resident slot
+# and that costs a FULL prompt reprocess — far worse than queueing.
+# =====================================================================
+LLAMA_BIN=/home/cogtek/llama.cpp/.worktrees/latest-master/build/bin/llama-server
+LLAMA_MODEL=/home/cogtek/Downloads/Qwen3.6-35B-A3B-UD-IQ4_XS.gguf
+LLAMA_MMPROJ=/home/cogtek/Downloads/mmproj-F16.gguf
+
+LLAMA_HOST=0.0.0.0
+LLAMA_PORT=8080
+LLAMA_CTX_SIZE=260000
+LLAMA_PARALLEL=2
+LLAMA_THREADS=16
+LLAMA_GPU_LAYERS=99
+
+# Qwen tolerates q4_0 KV well (unlike Gemma) and this model is large — 17.7 GB
+# of weights — so the cheap KV is what makes 260000 total fit at all.
+LLAMA_CACHE_TYPE_K=q4_0
+LLAMA_CACHE_TYPE_V=q4_0
+
+LLAMA_BATCH_SIZE=512
+LLAMA_UBATCH_SIZE=512
+
+# Speculative decoding OFF, deliberately. A Qwen3.6-35B-A3B-UD-IQ4_XS-MTP.gguf
+# exists on disk, but this is a MoE (A3B = ~3B active/token) and the Gemma 26B
+# A4B A/B on this same box showed MTP is a no-op on MoE while costing ~2 GB
+# (draft head + its own f16 draft KV). Do not wire it without re-measuring.
+# Draft-model spec also crashes the Qwen3.x hybrid-memory (GDN) seq_add path.
+LLAMA_ENABLE_SPEC_DECODING=false
+LLAMA_SPEC_TYPE=none
+LLAMA_DRAFT_MAX=3
+LLAMA_DRAFT_MIN=1
+LLAMA_DRAFT_P_MIN=0.75
+
+LLAMA_REPEAT_PENALTY=1.0
+LLAMA_CACHE_REUSE=256
+LLAMA_SLOT_SAVE_PATH=/home/cogtek/.cache/inference/kv-slots
+LLAMA_LOG_FILE=/home/cogtek/llama.cpp/llama-server.log
+LLAMA_N_PREDICT=16384
+
+# Qwen3.6 is ChatML with the qwen3_coder tool-call format; the proxy's parsers
+# are built around exactly this template. Do NOT switch it to "embedded".
+LLAMA_CHAT_TEMPLATE_FILE=/home/cogtek/dev/miller-tech/universal-agent-protocol/tools/agents/config/qwen3.5-enhanced.jinja
+
+# FLAG DRIFT: upstream renamed --checkpoint-every-n-tokens (old b8780 spelling,
+# which this profile used until 2026-07-28) to --checkpoint-min-step. An unknown
+# flag makes llama-server exit 1, and systemd then burns its restart budget
+# looping on it. Re-check every flag here against `llama-server --help` whenever
+# LLAMA_BIN moves.
+LLAMA_EXTRA_ARGS=--metrics --jinja --mlock --prio 2 --reasoning auto --reasoning-format deepseek --reasoning-budget 8000 --ctx-checkpoints 64 --checkpoint-min-step 2048 --cache-ram 16384