xinfer-ai 0.13.4 → 0.13.5

package/README.md

@@ -14,13 +14,21 @@
 | **⚡** | Fast | Native Flash Attention, FlashInfer, CUDA Graphs, continuous batching, prefix caching, PD disaggregation. Up to **197 tok/s** decode for `30B+` models on consumer GPUs |
 | **🪶** | Tiny footprint | Core scheduling + attention logic in **< 5 000 lines** of Rust |
 | **🌍** | Cross-platform | CUDA (Linux/Windows), Metal (macOS). Same binary, same API |
-| **🏭** | Production-ready | OpenAI/Anthropic-compatible APIs, built-in ChatGPT-style Web UI, MCP tool calling, structured outputs, embedding + tokenizer endpoints |
+| **🏭** | Production-ready | OpenAI/Anthropic-compatible APIs, built-in ChatGPT-style Web UI, MCP tool calling, structured outputs, embedding + tokenizer endpoints, multi-token prediction (MTP) |
 | **🗜️** | Aggressive KV compression | TurboQuant (`2–4 bit` KV cache) extends context up to **4.3×** with minimal quality loss. Run `30B+` MoE models with **millions of context** on single 24/32 GB GPUs |
 | **🔥** | V100 + NVFP4 | First-ever NVFP4 + low-bit KV cache on V100 — no hardware FP4 needed, coherent output on legacy GPUs |
 | **🐍** | Lightweight Python bindings | Optional PyO3 wheel when you need a Python entry point |
 
 ---
 
+## 📽️ Demo
+
+<p align="center">
+  <video src="https://guoqingbao.github.io/xinfer/assets/demo.mp4" width="600" controls></video>
+</p>
+
+---
+
 ## 🚀 Quick Start
 
 ### 📦 Install
@@ -55,6 +63,11 @@ xinfer --m /home/Qwen3.6-35B-A3B --d 0,1 --ui-server
 python3 -m xinfer.server --m Qwen/Qwen3.6-27B-FP8 --kvcache-dtype turbo4 --ui-server
 ```
 
+**MTP**
+```bash
+xinfer --w /home/Qwen3.6-35B-A3B --d 0,1 --ui-server --mtp 2
+```
+
 > **Tip:** Open `http://IP:8001` for the built-in chat UI, or use `http://IP:8000/v1/` as your API `Base URL`.
 
 ---
@@ -77,7 +90,7 @@ Add `--kvcache-dtype` to compress KV cache and extend context length:
 
 > Tested on **V100-32G**, **A100-40G**, **Hopper-80G** and **RTX 5090**
 
-| Model | Format | Size | Decoding Speed |
+| Model | Format | Size | Decoding Speed (without MTP) |
 |---|---|---|---|
 | Ministral-3-3B (**Multimodal**) | ISQ (BF16→Q4K) | 3B | **193.67** tokens/s |
 | Qwen3-VL-8B-Instruct (**Multimodal**) | Q8_0 | 8B | **112.51** tokens/s |
@@ -164,6 +177,9 @@ xinfer --m Qwen/Qwen3.6-35B-A3B-FP8 --kvcache-dtype fp8
 # 27B Dense + turbo4
 xinfer --m Qwen/Qwen3.6-27B-FP8 --kvcache-dtype turbo4
 
+# 26B Gemma4 (local model, occupy more kvcache with --kv-fraction)
+xinfer --w /data/gemma-4-26B-A4B-it --ui-server --port 9000 --kv-fraction 0.8
+
 # 30B MoE GGUF + turbo4
 xinfer --m unsloth/Qwen3-30B-A3B-Instruct-2507-GGUF \
   --f Qwen3-30B-A3B-Instruct-2507-Q4_K_M.gguf --kvcache-dtype turbo4
@@ -235,11 +251,17 @@ xinfer --m /path/Qwen3-4B --isq q6k
 <summary><b>GGUF models</b></summary>
 
 ```bash
-# Single GPU — GGUF
+# Multi-GPU GGUF (model ID + subfolder)
+xinfer --d 0,1,2,3 --m unsloth/Qwen3.5-122B-A10B-GGUF --f Q3_K_M
+
+# Remote GGUF — single-file model
 xinfer --m unsloth/Qwen3.5-27B-GGUF --f Qwen3.5-27B-Q4_K_M.gguf
 
-# Multi-GPU — GGUF
+# Local GGUF file (direct path)
 xinfer --d 0,1 --m /path/Qwen3-30B-A3B-Instruct-2507-Q4_K_M.gguf
+
+# Local GGUF folder
+xinfer --d 0,1,2,3 --m /path/to/model-GGUF/
 ```
 
 </details>
@@ -450,7 +472,7 @@ Constraint-based generation via llguidance — Lark grammars, regex, JSON Schema
 
 | Flag | Description |
 |---|---|
-| `--m` | Model source: HuggingFace model ID, local Safetensors directory, or local GGUF file |
+| `--m` | Model source: HuggingFace model ID, local Safetensors directory, local GGUF file, or local folder containing GGUF files |
 | `--w` | Legacy alias for local Safetensors directory; prefer `--m <local_dir>` |
 | `--f` | Local GGUF file when used alone; remote GGUF filename when paired with `--m <model_id>` |
 | `--d` | Device IDs (e.g. `--d 0,1`) |
@@ -477,6 +499,7 @@ Constraint-based generation via llguidance — Lark grammars, regex, JSON Schema
 | `--frequency-penalty` | Penalize frequent tokens (−2 to 2) |
 | `--mcp-config` | MCP servers JSON config |
 | `--mcp-command` / `--mcp-args` | Single MCP server command + args |
+| `--mtp`| Multi-token prediction, usage `--mtp 2` for two-token prediction per forward pass |
 
 ### Environment Variables
 
@@ -491,12 +514,6 @@ XINFER_NVFP4_FORCE_LUT=1 xinfer --m nvidia/Qwen3-30B-A3B-FP4 --ui-server
 
 ---
 
-## 📽️ Demo
-
-<video src="https://github.com/user-attachments/assets/7fc6aa0b-78ac-4323-923f-d761dd12857f" width="1000px"></video>
-
----
-
 ## 🛠️ Roadmap
 
 * [x] Batched inference (Metal)
@@ -534,6 +551,7 @@ XINFER_NVFP4_FORCE_LUT=1 xinfer --m nvidia/Qwen3-30B-A3B-FP4 --ui-server
 * [x] **MXFP4/NVFP4 Model Support**
 * [x] **Support Turboquant (4-bit, 3-bit) KvCache**
 * [ ] TentorRT-LLM
+* [x] **Multi-token Prediciton (MTP)**
 
 ---
 

package/docs/get_started.md

@@ -22,8 +22,13 @@ cargo build --release --features metal
 ## Model formats
 - **Safetensors (HF layout)**: `--m <hf_id>` for cached download, or `--m <local_dir>` for offline weights + configs. `--w <local_dir>` still works as a legacy alias.
 - **Safetensors (HF layout) + ISQ**: in-situ quantize into GGUF with args `--isq <q4k|q2k|q6k|...>`.
-- **GGUF**: `--m <local.gguf>` or `--f <local.gguf>` for a local file. Use `--m <hf_id> --f <remote_file.gguf>` for a GGUF file hosted in a Hugging Face repo.
-- **Vision-Language** (Qwen3-VL, Gemma3, Mistral3-VL): require image tokens; use `--ui-server` for uploads or send image_url/base64 in the request.
+- **GGUF**: multiple loading modes:
+  - `--m <local.gguf>` — load a single local GGUF file.
+  - `--m <local_dir>` — auto-detect the main GGUF file in a directory (picks the largest non-mmproj `.gguf`). Multi-shard files (`*-00001-of-00007.gguf`) and auxiliary files (`mmproj-*.gguf` for vision towers) are auto-discovered.
+  - `--f <local.gguf>` — load a single local GGUF file (alternative syntax).
+  - `--m <hf_id> --f <remote_file.gguf>` — download a GGUF file from a Hugging Face repo.
+  - `--m <hf_id> --f <subfolder>` — auto-detect all GGUF files in a HuggingFace repo subfolder (e.g. `--m unsloth/GLM-5.2-GGUF --f UD-Q2_K_XL`).
+- **Vision-Language** (Qwen3-VL, Qwen3.5-VL, Gemma3, Gemma4, Mistral3-VL): require image tokens; use `--ui-server` for uploads or send image_url/base64 in the request. For GGUF multimodal models, place the `mmproj-*.gguf` vision tower file in the same directory as the main model file — it will be auto-detected.
 
 
 ## 3) Run patterns (single host)
@@ -36,11 +41,19 @@ cargo build --release --features metal
   ```bash
   target/release/xinfer --m meta-llama/Llama-3-8b --max-model-len 32768 --ui-server
   ```
-- **GGUF quantized**  
+- **GGUF quantized (single file)**  
   ```bash
   target/release/xinfer --m /path/model-Q4_K_M.gguf --max-model-len 65536
   ```
-- **Remote GGUF quantized**  
+- **GGUF quantized (folder with auto-detection)**  
+  ```bash
+  target/release/xinfer --m /path/model-GGUF/ --ui-server
+  ```
+- **Remote GGUF quantized (multi-shard subfolder)**  
+  ```bash
+  target/release/xinfer --d 0,1,2,3 --m unsloth/GLM-5.2-GGUF --f UD-Q2_K_XL --ui-server
+  ```
+- **Remote GGUF quantized (single file)**  
   ```bash
   target/release/xinfer --m unsloth/Qwen3-0.6B-GGUF --f Qwen3-0.6B-Q4_K_M.gguf --ui-server
   ```

package/package.json

@@ -1,7 +1,7 @@
 {
   "name": "xinfer-ai",
-  "version": "0.13.4",
-  "assetVersion": "0.13.4",
+  "version": "0.13.5",
+  "assetVersion": "0.13.5",
   "description": "xInfer — a high-performance LLM inference engine in Rust with CUDA/Metal acceleration",
   "license": "MIT",
   "homepage": "https://github.com/guoqingbao/xinfer#readme",