xinfer-ai 0.14.5 → 0.14.6

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (3) hide show
  1. package/README.md +11 -2
  2. package/docs/moe_offload.md +136 -0
  3. package/package.json +2 -2
package/README.md CHANGED
@@ -78,6 +78,17 @@ xinfer --m Qwen/Qwen3.8-... --d 0,1 --ui-server \
78
78
 
79
79
  See [docs/speculative_decoding.md](docs/speculative_decoding.md) for details.
80
80
 
81
+ ### 🧩 MoE Expert Offloading
82
+
83
+ MoE expert offloading is supported on CUDA only. Run a large MoE model with expert weights offloaded to host RAM. The value is a fraction: `0.2` keeps 20% of each layer's experts on the GPU; `auto` selects the cache size automatically. Tensor parallelism requires NCCL.
84
+
85
+ ```bash
86
+ XINFER_MOE_OFFLOAD=0.2 xinfer \
87
+ --m Qwen/Qwen3.5-35B-A3B-FP8 --d 0,1 --ui-server
88
+ ```
89
+
90
+ See the [MoE expert offloading guide](docs/moe_offload.md) for configuration and TP details.
91
+
81
92
  > **Tip:** Open `http://IP:8001` for the built-in chat UI, or use `http://IP:8000/v1/` as your API `Base URL`.
82
93
 
83
94
  ---
@@ -94,8 +105,6 @@ Add `--kvcache-dtype` to compress KV cache and extend context length:
94
105
  | `turbo4` | **3.7×** | Best balance | SM70+ / Apple M1|
95
106
  | `turbo3` | **4.7×** | Max compression | SM70+ |
96
107
 
97
- ---
98
-
99
108
  ## 📈 Performance
100
109
 
101
110
  > Tested on **V100-32G**, **A100-40G**, **Hopper-80G** and **RTX 5090**
package/docs/moe_offload.md ADDED
@@ -0,0 +1,136 @@
1
+ # MoE expert offload
2
+
3
+ Run Mixture-of-Experts models whose weights do not fit in VRAM. The routed
4
+ experts stay in pinned host RAM; each MoE layer keeps a fraction of its experts
5
+ in GPU *slots*. Attention, dense layers, shared experts and the KV cache stay on
6
+ the GPU as usual.
7
+
8
+ This feature requires a CUDA build. Tensor parallelism (`--d 0,1`, etc.) also
9
+ requires NCCL; dense models do not enter the offload path and are unaffected
10
+ when the variable is unset.
11
+
12
+ ```bash
13
+ # Keep 20% of every layer's experts on GPU (at least one top-k decode step),
14
+ # and keep the remaining experts in host RAM. TP works rank-by-rank.
15
+ XINFER_MOE_OFFLOAD=0.2 xinfer \
16
+ --m Qwen/Qwen3.5-35B-A3B-FP8 --d 0,1 --max-model-len 8192 --ui-server
17
+
18
+ # Or size the per-layer GPU caches automatically from free VRAM.
19
+ XINFER_MOE_OFFLOAD=auto xinfer \
20
+ --m Qwen/Qwen3.5-35B-A3B-FP8 --d 0,1 --max-model-len 8192 --ui-server
21
+ ```
22
+
23
+ Placement never changes routing or precision: with greedy decoding the output
24
+ is byte-identical for every `XINFER_MOE_OFFLOAD` value.
25
+
26
+ ## How it works
27
+
28
+ - **Decode** — before each fused MoE GEMM, a CUDA kernel looks the routed experts
29
+ up in a per-layer LRU table kept on the GPU, assigns least-recently-used slots
30
+ to misses and rewrites expert ids to slot ids. A second kernel copies the
31
+ missing experts straight from device-mapped pinned host memory into their
32
+ slots. Both kernels have fixed launch shapes and read the miss count from
33
+ device memory, so decode stays inside CUDA graphs with no host sync.
34
+ - **Prefill** — when a call touches more distinct experts than the layer has
35
+ slots, the whole expert layer is copied (pinned async memcpy) into a staging
36
+ buffer, and the unmodified GEMM runs on it.
37
+ - The GEMM kernels are unchanged: they read `[num_slots, N, K]` slot tensors
38
+ instead of `[num_experts, N, K]`.
39
+
40
+ Design follows [FreeToken](https://github.com/FlashML-org/FreeToken) (Apache-2.0).
41
+
42
+ ## Options
43
+
44
+ | Variable | Meaning |
45
+ |---|---|
46
+ | `XINFER_MOE_OFFLOAD=auto` | After loading, grow every layer's slots evenly into the free VRAM, leaving the reserve below for the KV cache and workspaces (recommended) |
47
+ | `XINFER_MOE_OFFLOAD=<0..1>` | Fixed fraction of each layer's experts kept on the GPU; for example, `0.2` means 20% (at least `num_experts_per_tok`). Unset = no offload |
48
+ | `XINFER_MOE_OFFLOAD_RESERVE_GB` | VRAM left free by `auto` (default 2.5). Raise it for longer contexts |
49
+ | `XINFER_MOE_OFFLOAD_PROFILE=1` | Logs a per-MoE-call time split (launch gap, GPU wait, host, copy). Adds device syncs and disables CUDA graphs; diagnostics only |
50
+
51
+ More experts on the GPU means faster decode; more reserve means a longer
52
+ context. TurboQuant (`--kvcache-dtype turbo4|turbo3`) stretches the reserve
53
+ ~4x. Models with a large KV per token (e.g. Gemma 4) need `fp8` or `turbo*`.
54
+ MTP (`--num-speculative-tokens`) works but was slower in our tests.
55
+
56
+ For tensor-parallel inference, pass multiple devices with `--d 0,1` (or more
57
+ devices). Every rank must be able to read the model files locally. Each rank
58
+ loads and offloads its own expert shard, while NCCL continues to combine the
59
+ rank-local MoE results. The offload fraction applies independently to every
60
+ rank and every MoE layer.
61
+
62
+ ## Supported
63
+
64
+ - Formats:
65
+ - GGUF (`FusedMoeGGUF`, separate `ffn_gate_exps`/`ffn_up_exps`, quant types the
66
+ GGUF MoE kernel reads natively) and ISQ (`--isq`, `FusedMoeISQ`: experts are
67
+ quantized on the GPU at load, then moved to host RAM);
68
+ - NVFP4 ModelOpt safetensors (`FusedMoeNvfp4`);
69
+ - unquantized bf16/f16 (`FusedMoe`), block FP8 (`FusedMoeFp8`), MXFP4
70
+ (`FusedMoeMxfp4`) and GPTQ/AWQ/compressed-tensors WNA16 (`FusedMoeWNA16`): all
71
+ share one generic path that keeps every `[E, ...]` expert tensor (weights and
72
+ scales, any dtype) in host RAM.
73
+ - Not supported: W2-packed DeepSeek-V4 experts (fails with a clear error).
74
+ - CUDA single-GPU and tensor-parallel inference. NVFP4 on SM100+ (swizzled
75
+ scales) is not supported yet.
76
+ - Tested on real models: GGUF, NVFP4, bf16, ISQ, and FP8 MoE safetensors,
77
+ including Qwen3.5-35B-A3B-FP8 with two-rank tensor parallelism. MXFP4 and
78
+ WNA16 are covered by the generic path's bit-exact GEMM test.
79
+ - Host RAM must hold every expert (pinned).
80
+
81
+ ## Benchmarks
82
+
83
+ Reproduce with `bench/bench_moe_offload.py` (rate `0` = baseline without offload):
84
+
85
+ ```bash
86
+ python3 bench/bench_moe_offload.py --bin target/release/xinfer \
87
+ --model /models/Qwen3.6-35B-A3B-NVFP4 --rates 0,0.1,0.2,0.35,auto --out bench/qwen36
88
+ ```
89
+
90
+ It runs the OpenAI-compatible server, sends a warm-up and one streamed request
91
+ (greedy) and records time to first token, decode speed and inter-token latency
92
+ percentiles; `--prompt-tokens N` adds a long-context needle test. It writes a
93
+ Markdown table and the raw JSON next to `--out` (not committed).
94
+
95
+ RTX 3060 12 GB, PCIe 3.0 x16, Xeon E5-2650 v4, 64 GB DDR4-2400, Windows 11,
96
+ 200 decoded tokens, `--max-model-len 8192`. Without offload none of these
97
+ models runs on this card: the weights spill into the driver's sysmem fallback
98
+ and no VRAM is left for the KV cache.
99
+
100
+ **Qwen3.6-35B-A3B NVFP4** (23.4 GB of weights)
101
+
102
+ | GPU experts | KV dtype | KV tokens | TTFT | Decode tok/s | Inter-token p50 / p99 ms |
103
+ |---|---|---|---|---|---|
104
+ | none | — | does not run | | | |
105
+ | 10% | bf16 | 192k | 2.1 s | 21.5 | 46 / 70 |
106
+ | 20% | bf16 | 134k | 2.0 s | 26.4 | 38 / 51 |
107
+ | 35% | bf16 | 46k | 0.8 s | 28.6 | 35 / 44 |
108
+ | auto (31%) | bf16 | 58k | 1.3 s | 26.9 | 37 / 49 |
109
+ | 20% | turbo4 | 520k | 2.1 s | 23.3 | 41 / 68 |
110
+ | 35% | turbo3 | 201k | 1.0 s | 26.8 | 37 / 45 |
111
+
112
+ Long context: an 11.5k-token needle prompt at 20% prefills at 333 tok/s and the
113
+ needle is found. Greedy output is byte-identical for every GPU fraction.
114
+
115
+ **Gemma-4-26B-A4B NVFP4** (18.8 GB): 10% / 15% with `turbo4` → 16.7 / 19.8 tok/s
116
+ (27k / 16k KV tokens); with `fp8` at 10% → 18.9 tok/s.
117
+ **GLM-4.7-Flash NVFP4** (20.5 GB, no CUDA graphs in xinfer for this model):
118
+ 10% / 20% → 10.8 / 11.2 tok/s. Outputs identical across fractions for both.
119
+
120
+ **Qwen1.5-MoE-A2.7B-Chat bf16** (28.6 GB, 60 experts of 17 MB each), `auto` with
121
+ `XINFER_MOE_OFFLOAD_RESERVE_GB=1.5`:
122
+
123
+ | Experts | KV dtype | GPU experts | KV tokens | TTFT | Decode tok/s | Inter-token p50 / p99 ms |
124
+ |---|---|---|---|---|---|---|
125
+ | bf16 | fp8 | auto (~15%) | 8.4k | 2.5 s | 7.9 | 131 / 168 |
126
+ | bf16 | turbo4 | auto (~15%) | 12.5k | 2.4 s | 8.1 | 126 / 158 |
127
+ | ISQ `q8_0` | fp8 | auto (40%) | 14.8k | 1.9 s | 18.0 | 55 / 86 |
128
+ | ISQ `q8_0` | turbo4 | auto (40%) | 27.1k | 1.9 s | 20.5 | 49 / 74 |
129
+
130
+ bf16 experts are PCIe-bound (~1.4 GB of misses per token over PCIe 3.0);
131
+ `--isq q8_0` halves the expert size and fits 2.7x more experts on the GPU, so
132
+ decode is 2.5x faster. Q4K needs `moe_intermediate_size % 256 == 0` (1408 here)
133
+ and the GGUF MoE kernel has no Q4_0, so 4-bit ISQ is not available for this
134
+ model. bf16 greedy output is identical at 10% and auto (~15%).
135
+
136
+ Run-to-run variation on this machine is about ±10%.
package/package.json CHANGED
@@ -1,7 +1,7 @@
1
1
  {
2
2
  "name": "xinfer-ai",
3
- "version": "0.14.5",
4
- "assetVersion": "0.14.5",
3
+ "version": "0.14.6",
4
+ "assetVersion": "0.14.6",
5
5
  "description": "xInfer — a high-performance LLM inference engine in Rust with CUDA/Metal acceleration",
6
6
  "license": "MIT",
7
7
  "homepage": "https://github.com/guoqingbao/xinfer#readme",