xinfer-ai 0.13.4 → 0.13.6

package/README.md

@@ -14,13 +14,19 @@
 | **⚡** | Fast | Native Flash Attention, FlashInfer, CUDA Graphs, continuous batching, prefix caching, PD disaggregation. Up to **197 tok/s** decode for `30B+` models on consumer GPUs |
 | **🪶** | Tiny footprint | Core scheduling + attention logic in **< 5 000 lines** of Rust |
 | **🌍** | Cross-platform | CUDA (Linux/Windows), Metal (macOS). Same binary, same API |
-| **🏭** | Production-ready | OpenAI/Anthropic-compatible APIs, built-in ChatGPT-style Web UI, MCP tool calling, structured outputs, embedding + tokenizer endpoints |
+| **🏭** | Production-ready | OpenAI/Anthropic-compatible APIs, built-in ChatGPT-style Web UI, MCP tool calling, structured outputs, embedding + tokenizer endpoints, multi-token prediction (MTP) |
 | **🗜️** | Aggressive KV compression | TurboQuant (`2–4 bit` KV cache) extends context up to **4.3×** with minimal quality loss. Run `30B+` MoE models with **millions of context** on single 24/32 GB GPUs |
 | **🔥** | V100 + NVFP4 | First-ever NVFP4 + low-bit KV cache on V100 — no hardware FP4 needed, coherent output on legacy GPUs |
 | **🐍** | Lightweight Python bindings | Optional PyO3 wheel when you need a Python entry point |
 
 ---
 
+## 📽️ Demo
+
+<video src="pages/assets/demo.mp4" width="600" controls></video>
+
+---
+
 ## 🚀 Quick Start
 
 ### 📦 Install
@@ -55,6 +61,11 @@ xinfer --m /home/Qwen3.6-35B-A3B --d 0,1 --ui-server
 python3 -m xinfer.server --m Qwen/Qwen3.6-27B-FP8 --kvcache-dtype turbo4 --ui-server
 ```
 
+**MTP**
+```bash
+xinfer --w /home/Qwen3.6-35B-A3B --d 0,1 --ui-server --mtp 2
+```
+
 > **Tip:** Open `http://IP:8001` for the built-in chat UI, or use `http://IP:8000/v1/` as your API `Base URL`.
 
 ---
@@ -77,7 +88,7 @@ Add `--kvcache-dtype` to compress KV cache and extend context length:
 
 > Tested on **V100-32G**, **A100-40G**, **Hopper-80G** and **RTX 5090**
 
-| Model | Format | Size | Decoding Speed |
+| Model | Format | Size | Decoding Speed (without MTP) |
 |---|---|---|---|
 | Ministral-3-3B (**Multimodal**) | ISQ (BF16→Q4K) | 3B | **193.67** tokens/s |
 | Qwen3-VL-8B-Instruct (**Multimodal**) | Q8_0 | 8B | **112.51** tokens/s |
@@ -124,9 +135,10 @@ Add `--kvcache-dtype` to compress KV cache and extend context length:
 * ✅ Qwen3.5/3.6 Dense/MoE (27B, 35B, 122B, 397B, Multimodal model)
 * ✅ Mistral v1, v2
 * ✅ Mistral-3-VL Reasoning (3B, 8B, 14B, Multimodal model)
-* ✅ GLM4 (0414, **Not ChatGLM**)
+* ✅ GLM4 (0414)
 * ✅ GLM4 MoE (4.6/4.7)
 * ✅ GLM4.7 Flash
+* ✅ GLM 5.2 (DeepSeek V3.2 DSA architecture)
 * ✅ DeepSeek V3/R1/V3.2
 * ✅ Phi3 / Phi4 (Phi-3, Phi-4, Phi-4-mini, etc.)
 * ✅ Gemma3/**Gemma4** (Multimodal model)
@@ -164,6 +176,9 @@ xinfer --m Qwen/Qwen3.6-35B-A3B-FP8 --kvcache-dtype fp8
 # 27B Dense + turbo4
 xinfer --m Qwen/Qwen3.6-27B-FP8 --kvcache-dtype turbo4
 
+# 26B Gemma4 (local model, occupy more kvcache with --kv-fraction)
+xinfer --w /data/gemma-4-26B-A4B-it --ui-server --port 9000 --kv-fraction 0.8
+
 # 30B MoE GGUF + turbo4
 xinfer --m unsloth/Qwen3-30B-A3B-Instruct-2507-GGUF \
   --f Qwen3-30B-A3B-Instruct-2507-Q4_K_M.gguf --kvcache-dtype turbo4
@@ -202,6 +217,9 @@ xinfer --m olka-fi/Qwen3.5-4B-MXFP4
 # GGUF model (4-bit KvCache)
 xinfer --m unsloth/Qwen3.5-27B-GGUF --f Qwen3.5-27B-Q4_K_M.gguf --kvcache-dtype turbo4
 
+# Multi-shard 5.2 GGUF (auto-detected from HF subfolder)
+xinfer --d 0,1,2,3 --m unsloth/Qwen3.5-122B-A10B-GGUF --f Q3_K_M --kvcache-dtype fp8
+
 # FP8 on Metal
 xinfer --m Qwen/Qwen3.5-4B-FP8
 
@@ -213,6 +231,15 @@ xinfer --m GadflyII/GLM-4.7-Flash-NVFP4
 
 # Interactive CLI chat
 xinfer --i --m unsloth/Qwen3.5-27B-GGUF --f Qwen3.5-27B-Q4_K_M.gguf
+
+# Faster GDN prefill on Hopper with slight precision loss
+SM90_LOWER_PRECISION_GDN_PREFILL=1 xinfer --m Qwen/Qwen3.5-35B-A3B-FP8
+
+# MultiNode: GLM 5.2 (DeepSeek V3.2 architecture, FP8)
+# Master node
+xinfer --d 0,1,2,3,4,5,6,7 --m zai-org/GLM-5.2-FP8 --num-nodes 2 --node-rank 0 --master-addr 192.168.xxx.xxx
+# Other node(s)
+xinfer --d 0,1,2,3,4,5,6,7 --m zai-org/GLM-5.2-FP8 --num-nodes 2 --node-rank 1 --master-addr 192.168.xxx.xxx
 ```
 
 <details>
@@ -235,11 +262,17 @@ xinfer --m /path/Qwen3-4B --isq q6k
 <summary><b>GGUF models</b></summary>
 
 ```bash
-# Single GPU — GGUF
+# Multi-GPU GGUF (model ID + subfolder)
+xinfer --d 0,1,2,3 --m unsloth/Qwen3.5-122B-A10B-GGUF --f Q3_K_M
+
+# Remote GGUF — single-file model
 xinfer --m unsloth/Qwen3.5-27B-GGUF --f Qwen3.5-27B-Q4_K_M.gguf
 
-# Multi-GPU — GGUF
+# Local GGUF file (direct path)
 xinfer --d 0,1 --m /path/Qwen3-30B-A3B-Instruct-2507-Q4_K_M.gguf
+
+# Local GGUF folder
+xinfer --d 0,1,2,3 --m /path/to/Qwen3.5-122B-A10B-GGUF/Q3_K_M
 ```
 
 </details>
@@ -450,7 +483,7 @@ Constraint-based generation via llguidance — Lark grammars, regex, JSON Schema
 
 | Flag | Description |
 |---|---|
-| `--m` | Model source: HuggingFace model ID, local Safetensors directory, or local GGUF file |
+| `--m` | Model source: HuggingFace model ID, local Safetensors directory, local GGUF file, or local folder containing GGUF files |
 | `--w` | Legacy alias for local Safetensors directory; prefer `--m <local_dir>` |
 | `--f` | Local GGUF file when used alone; remote GGUF filename when paired with `--m <model_id>` |
 | `--d` | Device IDs (e.g. `--d 0,1`) |
@@ -477,23 +510,25 @@ Constraint-based generation via llguidance — Lark grammars, regex, JSON Schema
 | `--frequency-penalty` | Penalize frequent tokens (−2 to 2) |
 | `--mcp-config` | MCP servers JSON config |
 | `--mcp-command` / `--mcp-args` | Single MCP server command + args |
+| `--mtp`| Multi-token prediction, usage `--mtp 2` for two-token prediction per forward pass |
 
 ### Environment Variables
 
 | Variable | Description |
 |---|---|
 | `XINFER_NVFP4_FORCE_LUT=1` | Force software NVFP4 decode to use the LUT-based dequantization path (higher precision) instead of hardware FP4 intrinsics on Blackwell (SM100+). Useful when decode precision matters more than peak throughput. |
+| `XINFER_SSE_BUFFER_SIZE=256` | Size of the bounded SSE streaming buffer per client connection (default: 256). Increase for slow network proxies or high-throughput models. |
+| `SM90_LOWER_PRECISION_GDN_PREFILL=1` | Enable the FlashInfer SM90 persistent kernel for GatedDeltaNet (GDN) prefill on Hopper GPUs (SM90). Delivers faster prefill speedup for Qwen3.5/3.6, with a slight precision trade-off. |
 
 **Example (Blackwell with high-precision NVFP4 decode):**
 ```bash
 XINFER_NVFP4_FORCE_LUT=1 xinfer --m nvidia/Qwen3-30B-A3B-FP4 --ui-server
 ```
 
----
-
-## 📽️ Demo
-
-<video src="https://github.com/user-attachments/assets/7fc6aa0b-78ac-4323-923f-d761dd12857f" width="1000px"></video>
+**Example (Hopper with faster GDN prefill):**
+```bash
+SM90_LOWER_PRECISION_GDN_PREFILL=1 xinfer --m Qwen/Qwen3.5-35B-A3B-FP8 --ui-server
+```
 
 ---
 
@@ -518,7 +553,8 @@ XINFER_NVFP4_FORCE_LUT=1 xinfer --m nvidia/Qwen3-30B-A3B-FP4 --ui-server
 * [x] FP8 KV Cache (with FlashInfer, SM80+)
 * [x] TurboQuant KV Cache (2-4 bit compression with WHT rotation)
 * [x] FP8 Models (CUDA: MoE, Dense; Metal: Dense)
-* [ ] Additional model support (Kimi K2, GLM 5.1 etc.)
+* [x] GLM 5.2 (DeepSeek V3.2 + DSA)
+* [ ] Additional model support (Kimi K2, etc.)
 * [x] CPU KV Cache Offloading
 * [x] Prefill-decode Disaggregation (CUDA)
 * [x] Prefill-decode Disaggregation (Metal)
@@ -534,6 +570,7 @@ XINFER_NVFP4_FORCE_LUT=1 xinfer --m nvidia/Qwen3-30B-A3B-FP4 --ui-server
 * [x] **MXFP4/NVFP4 Model Support**
 * [x] **Support Turboquant (4-bit, 3-bit) KvCache**
 * [ ] TentorRT-LLM
+* [x] **Multi-token Prediciton (MTP)**
 
 ---
 

package/docs/get_started.md

@@ -22,8 +22,13 @@ cargo build --release --features metal
 ## Model formats
 - **Safetensors (HF layout)**: `--m <hf_id>` for cached download, or `--m <local_dir>` for offline weights + configs. `--w <local_dir>` still works as a legacy alias.
 - **Safetensors (HF layout) + ISQ**: in-situ quantize into GGUF with args `--isq <q4k|q2k|q6k|...>`.
-- **GGUF**: `--m <local.gguf>` or `--f <local.gguf>` for a local file. Use `--m <hf_id> --f <remote_file.gguf>` for a GGUF file hosted in a Hugging Face repo.
-- **Vision-Language** (Qwen3-VL, Gemma3, Mistral3-VL): require image tokens; use `--ui-server` for uploads or send image_url/base64 in the request.
+- **GGUF**: multiple loading modes:
+  - `--m <local.gguf>` — load a single local GGUF file.
+  - `--m <local_dir>` — auto-detect the main GGUF file in a directory (picks the largest non-mmproj `.gguf`). Multi-shard files (`*-00001-of-00007.gguf`) and auxiliary files (`mmproj-*.gguf` for vision towers) are auto-discovered.
+  - `--f <local.gguf>` — load a single local GGUF file (alternative syntax).
+  - `--m <hf_id> --f <remote_file.gguf>` — download a GGUF file from a Hugging Face repo.
+  - `--m <hf_id> --f <subfolder>` — auto-detect all GGUF files in a HuggingFace repo subfolder (e.g. `--m unsloth/GLM-5.2-GGUF --f UD-Q2_K_XL`).
+- **Vision-Language** (Qwen3-VL, Qwen3.5-VL, Gemma3, Gemma4, Mistral3-VL): require image tokens; use `--ui-server` for uploads or send image_url/base64 in the request. For GGUF multimodal models, place the `mmproj-*.gguf` vision tower file in the same directory as the main model file — it will be auto-detected.
 
 
 ## 3) Run patterns (single host)
@@ -36,11 +41,19 @@ cargo build --release --features metal
   ```bash
   target/release/xinfer --m meta-llama/Llama-3-8b --max-model-len 32768 --ui-server
   ```
-- **GGUF quantized**  
+- **GGUF quantized (single file)**  
   ```bash
   target/release/xinfer --m /path/model-Q4_K_M.gguf --max-model-len 65536
   ```
-- **Remote GGUF quantized**  
+- **GGUF quantized (folder with auto-detection)**  
+  ```bash
+  target/release/xinfer --m /path/GLM-5.2-GGUF/ --ui-server
+  ```
+- **Remote GGUF quantized (multi-shard subfolder)**  
+  ```bash
+  target/release/xinfer --d 0,1,2,3 --m unsloth/GLM-5.2-GGUF --f UD-Q2_K_XL --ui-server
+  ```
+- **Remote GGUF quantized (single file)**  
   ```bash
   target/release/xinfer --m unsloth/Qwen3-0.6B-GGUF --f Qwen3-0.6B-Q4_K_M.gguf --ui-server
   ```

package/package.json

@@ -1,7 +1,7 @@
 {
   "name": "xinfer-ai",
-  "version": "0.13.4",
-  "assetVersion": "0.13.4",
+  "version": "0.13.6",
+  "assetVersion": "0.13.6",
   "description": "xInfer — a high-performance LLM inference engine in Rust with CUDA/Metal acceleration",
   "license": "MIT",
   "homepage": "https://github.com/guoqingbao/xinfer#readme",