xinfer-ai 0.14.5 → 0.14.7

package/README.md

@@ -78,6 +78,17 @@ xinfer --m Qwen/Qwen3.8-... --d 0,1 --ui-server \
 
 See [docs/speculative_decoding.md](docs/speculative_decoding.md) for details.
 
+### 🧩 MoE Expert Offloading
+
+MoE expert offloading is supported on CUDA only. Run a large MoE model with expert weights offloaded to host RAM. The value is a fraction: `0.2` keeps 20% of each layer's experts on the GPU; `auto` selects the cache size automatically. Tensor parallelism requires NCCL.
+
+```bash
+XINFER_MOE_OFFLOAD=0.2 xinfer \
+  --m Qwen/Qwen3.5-35B-A3B-FP8 --d 0,1 --ui-server
+```
+
+See the [MoE expert offloading guide](docs/moe_offload.md) for configuration and TP details.
+
 > **Tip:** Open `http://IP:8001` for the built-in chat UI, or use `http://IP:8000/v1/` as your API `Base URL`.
 
 ---
@@ -94,8 +105,6 @@ Add `--kvcache-dtype` to compress KV cache and extend context length:
 | `turbo4` | **3.7×** | Best balance | SM70+ / Apple M1|
 | `turbo3` | **4.7×** | Max compression | SM70+ |
 
----
-
 ## 📈 Performance
 
 > Tested on **V100-32G**, **A100-40G**, **Hopper-80G** and **RTX 5090**
@@ -105,17 +114,17 @@ Add `--kvcache-dtype` to compress KV cache and extend context length:
 | Ministral-3-3B (**Multimodal**) | ISQ (BF16→Q4K) | 3B | **193.67** tokens/s |
 | Qwen3-VL-8B-Instruct (**Multimodal**) | Q8_0 | 8B | **112.51** tokens/s |
 | Llama-3.1-8B | ISQ (BF16→Q4K) | 8B | **133.10** tokens/s |
-| DeepSeek-R1-0528-Qwen3-8B | Q4_K_M | 8B | **139.25** tokens/s |
-| GLM-4-9B-0414 | Q4_K_M | 9B | **77.48** tokens/s |
-| QwQ-32B | Q4_K_M | 32B | **46.02** tokens/s |
+| DeepSeek-R1-0528-Qwen3-8B | Q4_K_M | 8B | **161.60** tokens/s |
+| GLM-4-9B-0414 | Q4_K_M | 9B | **92.10** tokens/s |
+| QwQ-32B | Q4_K_M | 32B | **55.65** tokens/s |
 | **Qwen3-30B-A3B** | NVFP4 | **30B (MoE)** | **197.29** tokens/s (**RTX 5090**) |
 | **Qwen3-30B-A3B** | NVFP4 | **30B (MoE)** | **72.86** tokens/s (**V100, Software FP4**) |
 | **Qwen3-Next-80B** | AWQ | **80B (MoE)** | **90** tokens/s (**Hopper**) |
-| **Qwen3.5/3.6/3.8 27B** (**Multimodal**) | Q4_K_M/IQ4 | **27B (Dense)** | **58** tokens/s |
-| **Qwen3.5/3.6/3.8 27B** | FP8 | **27B (Dense)** | **45** tokens/s (**Hopper**) |
+| **Qwen3.5/3.6/3.8 27B** (**Multimodal**) | Q4_K_M/IQ4 | **27B (Dense)** | **61.9** tokens/s |
+| **Qwen3.5/3.6/3.8 27B** | FP8 | **27B (Dense)** | **52.2** tokens/s (**Hopper**) |
 | **Qwen3.6-35B-A3B** (**Multimodal**) | FP8 | **35B (MoE)** | **120** tokens/s (**Hopper**) |
 | **GLM4.7 Flash** | NVFP4 | **30B (MoE)** | **79** tokens/s (**Hopper, Software FP4**) |
-| **Gemma4-31B** | ISQ (BF16→Q4K) | **31B (Dense)** | **47** tokens/s (**Hopper**) |
+| **Gemma4-31B** | ISQ (BF16→Q4K) | **31B (Dense)** | **55** tokens/s (**Hopper**) |
 | **Gemma4-26B-A4B** | NVFP4 | **26B (MoE)** | **137.23** tokens/s (**RTX 5090**) |
 | **MiniMax-M2.5** | NVFP4 | **229B (MoE)** | **64.50** tokens/s (**Hopper, Software FP4, TP=2**) |
 

package/docs/moe_offload.md

@@ -0,0 +1,136 @@
+# MoE expert offload
+
+Run Mixture-of-Experts models whose weights do not fit in VRAM. The routed
+experts stay in pinned host RAM; each MoE layer keeps a fraction of its experts
+in GPU *slots*. Attention, dense layers, shared experts and the KV cache stay on
+the GPU as usual.
+
+This feature requires a CUDA build. Tensor parallelism (`--d 0,1`, etc.) also
+requires NCCL; dense models do not enter the offload path and are unaffected
+when the variable is unset.
+
+```bash
+# Keep 20% of every layer's experts on GPU (at least one top-k decode step),
+# and keep the remaining experts in host RAM. TP works rank-by-rank.
+XINFER_MOE_OFFLOAD=0.2 xinfer \
+    --m Qwen/Qwen3.5-35B-A3B-FP8 --d 0,1 --max-model-len 8192 --ui-server
+
+# Or size the per-layer GPU caches automatically from free VRAM.
+XINFER_MOE_OFFLOAD=auto xinfer \
+    --m Qwen/Qwen3.5-35B-A3B-FP8 --d 0,1 --max-model-len 8192 --ui-server
+```
+
+Placement never changes routing or precision: with greedy decoding the output
+is byte-identical for every `XINFER_MOE_OFFLOAD` value.
+
+## How it works
+
+- **Decode** — before each fused MoE GEMM, a CUDA kernel looks the routed experts
+  up in a per-layer LRU table kept on the GPU, assigns least-recently-used slots
+  to misses and rewrites expert ids to slot ids. A second kernel copies the
+  missing experts straight from device-mapped pinned host memory into their
+  slots. Both kernels have fixed launch shapes and read the miss count from
+  device memory, so decode stays inside CUDA graphs with no host sync.
+- **Prefill** — when a call touches more distinct experts than the layer has
+  slots, the whole expert layer is copied (pinned async memcpy) into a staging
+  buffer, and the unmodified GEMM runs on it.
+- The GEMM kernels are unchanged: they read `[num_slots, N, K]` slot tensors
+  instead of `[num_experts, N, K]`.
+
+Design follows [FreeToken](https://github.com/FlashML-org/FreeToken) (Apache-2.0).
+
+## Options
+
+| Variable | Meaning |
+|---|---|
+| `XINFER_MOE_OFFLOAD=auto` | After loading, grow every layer's slots evenly into the free VRAM, leaving the reserve below for the KV cache and workspaces (recommended) |
+| `XINFER_MOE_OFFLOAD=<0..1>` | Fixed fraction of each layer's experts kept on the GPU; for example, `0.2` means 20% (at least `num_experts_per_tok`). Unset = no offload |
+| `XINFER_MOE_OFFLOAD_RESERVE_GB` | VRAM left free by `auto` (default 2.5). Raise it for longer contexts |
+| `XINFER_MOE_OFFLOAD_PROFILE=1` | Logs a per-MoE-call time split (launch gap, GPU wait, host, copy). Adds device syncs and disables CUDA graphs; diagnostics only |
+
+More experts on the GPU means faster decode; more reserve means a longer
+context. TurboQuant (`--kvcache-dtype turbo4|turbo3`) stretches the reserve
+~4x. Models with a large KV per token (e.g. Gemma 4) need `fp8` or `turbo*`.
+MTP (`--num-speculative-tokens`) works but was slower in our tests.
+
+For tensor-parallel inference, pass multiple devices with `--d 0,1` (or more
+devices). Every rank must be able to read the model files locally. Each rank
+loads and offloads its own expert shard, while NCCL continues to combine the
+rank-local MoE results. The offload fraction applies independently to every
+rank and every MoE layer.
+
+## Supported
+
+- Formats:
+  - GGUF (`FusedMoeGGUF`, separate `ffn_gate_exps`/`ffn_up_exps`, quant types the
+    GGUF MoE kernel reads natively) and ISQ (`--isq`, `FusedMoeISQ`: experts are
+    quantized on the GPU at load, then moved to host RAM);
+  - NVFP4 ModelOpt safetensors (`FusedMoeNvfp4`);
+  - unquantized bf16/f16 (`FusedMoe`), block FP8 (`FusedMoeFp8`), MXFP4
+    (`FusedMoeMxfp4`) and GPTQ/AWQ/compressed-tensors WNA16 (`FusedMoeWNA16`): all
+    share one generic path that keeps every `[E, ...]` expert tensor (weights and
+    scales, any dtype) in host RAM.
+- Not supported: W2-packed DeepSeek-V4 experts (fails with a clear error).
+- CUDA single-GPU and tensor-parallel inference. NVFP4 on SM100+ (swizzled
+  scales) is not supported yet.
+- Tested on real models: GGUF, NVFP4, bf16, ISQ, and FP8 MoE safetensors,
+  including Qwen3.5-35B-A3B-FP8 with two-rank tensor parallelism. MXFP4 and
+  WNA16 are covered by the generic path's bit-exact GEMM test.
+- Host RAM must hold every expert (pinned).
+
+## Benchmarks
+
+Reproduce with `bench/bench_moe_offload.py` (rate `0` = baseline without offload):
+
+```bash
+python3 bench/bench_moe_offload.py --bin target/release/xinfer \
+    --model /models/Qwen3.6-35B-A3B-NVFP4 --rates 0,0.1,0.2,0.35,auto --out bench/qwen36
+```
+
+It runs the OpenAI-compatible server, sends a warm-up and one streamed request
+(greedy) and records time to first token, decode speed and inter-token latency
+percentiles; `--prompt-tokens N` adds a long-context needle test. It writes a
+Markdown table and the raw JSON next to `--out` (not committed).
+
+RTX 3060 12 GB, PCIe 3.0 x16, Xeon E5-2650 v4, 64 GB DDR4-2400, Windows 11,
+200 decoded tokens, `--max-model-len 8192`. Without offload none of these
+models runs on this card: the weights spill into the driver's sysmem fallback
+and no VRAM is left for the KV cache.
+
+**Qwen3.6-35B-A3B NVFP4** (23.4 GB of weights)
+
+| GPU experts | KV dtype | KV tokens | TTFT | Decode tok/s | Inter-token p50 / p99 ms |
+|---|---|---|---|---|---|
+| none | — | does not run | | | |
+| 10% | bf16 | 192k | 2.1 s | 21.5 | 46 / 70 |
+| 20% | bf16 | 134k | 2.0 s | 26.4 | 38 / 51 |
+| 35% | bf16 | 46k | 0.8 s | 28.6 | 35 / 44 |
+| auto (31%) | bf16 | 58k | 1.3 s | 26.9 | 37 / 49 |
+| 20% | turbo4 | 520k | 2.1 s | 23.3 | 41 / 68 |
+| 35% | turbo3 | 201k | 1.0 s | 26.8 | 37 / 45 |
+
+Long context: an 11.5k-token needle prompt at 20% prefills at 333 tok/s and the
+needle is found. Greedy output is byte-identical for every GPU fraction.
+
+**Gemma-4-26B-A4B NVFP4** (18.8 GB): 10% / 15% with `turbo4` → 16.7 / 19.8 tok/s
+(27k / 16k KV tokens); with `fp8` at 10% → 18.9 tok/s.
+**GLM-4.7-Flash NVFP4** (20.5 GB, no CUDA graphs in xinfer for this model):
+10% / 20% → 10.8 / 11.2 tok/s. Outputs identical across fractions for both.
+
+**Qwen1.5-MoE-A2.7B-Chat bf16** (28.6 GB, 60 experts of 17 MB each), `auto` with
+`XINFER_MOE_OFFLOAD_RESERVE_GB=1.5`:
+
+| Experts | KV dtype | GPU experts | KV tokens | TTFT | Decode tok/s | Inter-token p50 / p99 ms |
+|---|---|---|---|---|---|---|
+| bf16 | fp8 | auto (~15%) | 8.4k | 2.5 s | 7.9 | 131 / 168 |
+| bf16 | turbo4 | auto (~15%) | 12.5k | 2.4 s | 8.1 | 126 / 158 |
+| ISQ `q8_0` | fp8 | auto (40%) | 14.8k | 1.9 s | 18.0 | 55 / 86 |
+| ISQ `q8_0` | turbo4 | auto (40%) | 27.1k | 1.9 s | 20.5 | 49 / 74 |
+
+bf16 experts are PCIe-bound (~1.4 GB of misses per token over PCIe 3.0);
+`--isq q8_0` halves the expert size and fits 2.7x more experts on the GPU, so
+decode is 2.5x faster. Q4K needs `moe_intermediate_size % 256 == 0` (1408 here)
+and the GGUF MoE kernel has no Q4_0, so 4-bit ISQ is not available for this
+model. bf16 greedy output is identical at 10% and auto (~15%).
+
+Run-to-run variation on this machine is about ±10%.

package/package.json

@@ -1,7 +1,7 @@
 {
   "name": "xinfer-ai",
-  "version": "0.14.5",
-  "assetVersion": "0.14.5",
+  "version": "0.14.7",
+  "assetVersion": "0.14.7",
   "description": "xInfer — a high-performance LLM inference engine in Rust with CUDA/Metal acceleration",
   "license": "MIT",
   "homepage": "https://github.com/guoqingbao/xinfer#readme",