xinfer-ai 0.13.3 → 0.13.5

package/README.md

@@ -14,13 +14,21 @@
 | **⚡** | Fast | Native Flash Attention, FlashInfer, CUDA Graphs, continuous batching, prefix caching, PD disaggregation. Up to **197 tok/s** decode for `30B+` models on consumer GPUs |
 | **🪶** | Tiny footprint | Core scheduling + attention logic in **< 5 000 lines** of Rust |
 | **🌍** | Cross-platform | CUDA (Linux/Windows), Metal (macOS). Same binary, same API |
-| **🏭** | Production-ready | OpenAI/Anthropic-compatible APIs, built-in ChatGPT-style Web UI, MCP tool calling, structured outputs, embedding + tokenizer endpoints |
+| **🏭** | Production-ready | OpenAI/Anthropic-compatible APIs, built-in ChatGPT-style Web UI, MCP tool calling, structured outputs, embedding + tokenizer endpoints, multi-token prediction (MTP) |
 | **🗜️** | Aggressive KV compression | TurboQuant (`2–4 bit` KV cache) extends context up to **4.3×** with minimal quality loss. Run `30B+` MoE models with **millions of context** on single 24/32 GB GPUs |
 | **🔥** | V100 + NVFP4 | First-ever NVFP4 + low-bit KV cache on V100 — no hardware FP4 needed, coherent output on legacy GPUs |
 | **🐍** | Lightweight Python bindings | Optional PyO3 wheel when you need a Python entry point |
 
 ---
 
+## 📽️ Demo
+
+<p align="center">
+  <video src="https://guoqingbao.github.io/xinfer/assets/demo.mp4" width="600" controls></video>
+</p>
+
+---
+
 ## 🚀 Quick Start
 
 ### 📦 Install
@@ -55,6 +63,11 @@ xinfer --m /home/Qwen3.6-35B-A3B --d 0,1 --ui-server
 python3 -m xinfer.server --m Qwen/Qwen3.6-27B-FP8 --kvcache-dtype turbo4 --ui-server
 ```
 
+**MTP**
+```bash
+xinfer --w /home/Qwen3.6-35B-A3B --d 0,1 --ui-server --mtp 2
+```
+
 > **Tip:** Open `http://IP:8001` for the built-in chat UI, or use `http://IP:8000/v1/` as your API `Base URL`.
 
 ---
@@ -77,7 +90,7 @@ Add `--kvcache-dtype` to compress KV cache and extend context length:
 
 > Tested on **V100-32G**, **A100-40G**, **Hopper-80G** and **RTX 5090**
 
-| Model | Format | Size | Decoding Speed |
+| Model | Format | Size | Decoding Speed (without MTP) |
 |---|---|---|---|
 | Ministral-3-3B (**Multimodal**) | ISQ (BF16→Q4K) | 3B | **193.67** tokens/s |
 | Qwen3-VL-8B-Instruct (**Multimodal**) | Q8_0 | 8B | **112.51** tokens/s |
@@ -164,6 +177,9 @@ xinfer --m Qwen/Qwen3.6-35B-A3B-FP8 --kvcache-dtype fp8
 # 27B Dense + turbo4
 xinfer --m Qwen/Qwen3.6-27B-FP8 --kvcache-dtype turbo4
 
+# 26B Gemma4 (local model, occupy more kvcache with --kv-fraction)
+xinfer --w /data/gemma-4-26B-A4B-it --ui-server --port 9000 --kv-fraction 0.8
+
 # 30B MoE GGUF + turbo4
 xinfer --m unsloth/Qwen3-30B-A3B-Instruct-2507-GGUF \
   --f Qwen3-30B-A3B-Instruct-2507-Q4_K_M.gguf --kvcache-dtype turbo4
@@ -235,11 +251,17 @@ xinfer --m /path/Qwen3-4B --isq q6k
 <summary><b>GGUF models</b></summary>
 
 ```bash
-# Single GPU — GGUF
+# Multi-GPU GGUF (model ID + subfolder)
+xinfer --d 0,1,2,3 --m unsloth/Qwen3.5-122B-A10B-GGUF --f Q3_K_M
+
+# Remote GGUF — single-file model
 xinfer --m unsloth/Qwen3.5-27B-GGUF --f Qwen3.5-27B-Q4_K_M.gguf
 
-# Multi-GPU — GGUF
+# Local GGUF file (direct path)
 xinfer --d 0,1 --m /path/Qwen3-30B-A3B-Instruct-2507-Q4_K_M.gguf
+
+# Local GGUF folder
+xinfer --d 0,1,2,3 --m /path/to/model-GGUF/
 ```
 
 </details>
@@ -338,6 +360,24 @@ See [more Python examples →](python/ReadMe.md)
 
 ---
 
+## 🌐 Multi-Node Tensor Parallelism
+
+Distribute inference across multiple machines using TCP-based NCCL bootstrap. No MPI required.
+
+```bash
+# Node 0 (master, 192.168.1.100): scheduler + API
+xinfer --d 0,1,2,3 --m /data/DeepSeek-R1/ \
+  --num-nodes 2 --node-rank 0 --master-addr 192.168.1.100 --ui-server
+
+# Node 1 (worker, 192.168.1.101): forward-only daemon
+xinfer --d 0,1,2,3 --m /data/DeepSeek-R1/ \
+  --num-nodes 2 --node-rank 1 --master-addr 192.168.1.100
+```
+
+All nodes must have model weights locally and be TCP-reachable on `--master-port` (default 29500). See [Get Started](docs/get_started.md) for details.
+
+---
+
 ## 🔀 Prefill-Decode Disaggregation
 
 Split prefill (prompt processing) and decode (token generation) across GPUs or machines. Eliminates decode stalls during long-context prefilling. PD Server and PD Client must use **same** KvCache type (`--kvcache-dtype`). API request(s) must send to PD Client and the PD Server only process internal prefill requests sent from PD Client.
@@ -432,12 +472,12 @@ Constraint-based generation via llguidance — Lark grammars, regex, JSON Schema
 
 | Flag | Description |
 |---|---|
-| `--m` | Model source: HuggingFace model ID, local Safetensors directory, or local GGUF file |
+| `--m` | Model source: HuggingFace model ID, local Safetensors directory, local GGUF file, or local folder containing GGUF files |
 | `--w` | Legacy alias for local Safetensors directory; prefer `--m <local_dir>` |
 | `--f` | Local GGUF file when used alone; remote GGUF filename when paired with `--m <model_id>` |
 | `--d` | Device IDs (e.g. `--d 0,1`) |
 | `--ui-server` | API server + built-in ChatGPT-style web UI |
-| `--server` | API server only (no web UI) |
+| `--server` | API server. Bare: `0.0.0.0:8000`; or `host[:port]`, `file:///path`, `socket:///path`, `unix:///path` |
 | `--i` | Interactive CLI chat |
 | `--isq` | On-the-fly quantization: `q2k`, `q3k`, `q4k`, `q5k`, `q6k`, `q8_0` |
 | `--kvcache-dtype` | KV cache quantization: `fp8`, `turbo8`, `turbo4`, `turbo3` |
@@ -459,6 +499,7 @@ Constraint-based generation via llguidance — Lark grammars, regex, JSON Schema
 | `--frequency-penalty` | Penalize frequent tokens (−2 to 2) |
 | `--mcp-config` | MCP servers JSON config |
 | `--mcp-command` / `--mcp-args` | Single MCP server command + args |
+| `--mtp`| Multi-token prediction, usage `--mtp 2` for two-token prediction per forward pass |
 
 ### Environment Variables
 
@@ -473,12 +514,6 @@ XINFER_NVFP4_FORCE_LUT=1 xinfer --m nvidia/Qwen3-30B-A3B-FP4 --ui-server
 
 ---
 
-## 📽️ Demo
-
-<video src="https://github.com/user-attachments/assets/7fc6aa0b-78ac-4323-923f-d761dd12857f" width="1000px"></video>
-
----
-
 ## 🛠️ Roadmap
 
 * [x] Batched inference (Metal)
@@ -488,6 +523,7 @@ XINFER_NVFP4_FORCE_LUT=1 xinfer --m nvidia/Qwen3-30B-A3B-FP4 --ui-server
 * [x] OpenAI-compatible API (streaming support)
 * [x] Continuous batching
 * [x] Multi-gpu inference (Safetensors, GPTQ, AWQ, GGUF)
+* [x] Multi-node tensor parallelism (TCP-based NCCL bootstrap, no MPI required)
 * [x] Speedup prompt processing on Metal/macOS
 * [x] Chunked Prefill
 * [x] Prefix cache (available on `CUDA` when `prefix-cache` enabled)
@@ -515,6 +551,7 @@ XINFER_NVFP4_FORCE_LUT=1 xinfer --m nvidia/Qwen3-30B-A3B-FP4 --ui-server
 * [x] **MXFP4/NVFP4 Model Support**
 * [x] **Support Turboquant (4-bit, 3-bit) KvCache**
 * [ ] TentorRT-LLM
+* [x] **Multi-token Prediciton (MTP)**
 
 ---
 

package/docs/get_started.md

@@ -22,8 +22,13 @@ cargo build --release --features metal
 ## Model formats
 - **Safetensors (HF layout)**: `--m <hf_id>` for cached download, or `--m <local_dir>` for offline weights + configs. `--w <local_dir>` still works as a legacy alias.
 - **Safetensors (HF layout) + ISQ**: in-situ quantize into GGUF with args `--isq <q4k|q2k|q6k|...>`.
-- **GGUF**: `--m <local.gguf>` or `--f <local.gguf>` for a local file. Use `--m <hf_id> --f <remote_file.gguf>` for a GGUF file hosted in a Hugging Face repo.
-- **Vision-Language** (Qwen3-VL, Gemma3, Mistral3-VL): require image tokens; use `--ui-server` for uploads or send image_url/base64 in the request.
+- **GGUF**: multiple loading modes:
+  - `--m <local.gguf>` — load a single local GGUF file.
+  - `--m <local_dir>` — auto-detect the main GGUF file in a directory (picks the largest non-mmproj `.gguf`). Multi-shard files (`*-00001-of-00007.gguf`) and auxiliary files (`mmproj-*.gguf` for vision towers) are auto-discovered.
+  - `--f <local.gguf>` — load a single local GGUF file (alternative syntax).
+  - `--m <hf_id> --f <remote_file.gguf>` — download a GGUF file from a Hugging Face repo.
+  - `--m <hf_id> --f <subfolder>` — auto-detect all GGUF files in a HuggingFace repo subfolder (e.g. `--m unsloth/GLM-5.2-GGUF --f UD-Q2_K_XL`).
+- **Vision-Language** (Qwen3-VL, Qwen3.5-VL, Gemma3, Gemma4, Mistral3-VL): require image tokens; use `--ui-server` for uploads or send image_url/base64 in the request. For GGUF multimodal models, place the `mmproj-*.gguf` vision tower file in the same directory as the main model file — it will be auto-detected.
 
 
 ## 3) Run patterns (single host)
@@ -36,11 +41,19 @@ cargo build --release --features metal
   ```bash
   target/release/xinfer --m meta-llama/Llama-3-8b --max-model-len 32768 --ui-server
   ```
-- **GGUF quantized**  
+- **GGUF quantized (single file)**  
   ```bash
   target/release/xinfer --m /path/model-Q4_K_M.gguf --max-model-len 65536
   ```
-- **Remote GGUF quantized**  
+- **GGUF quantized (folder with auto-detection)**  
+  ```bash
+  target/release/xinfer --m /path/model-GGUF/ --ui-server
+  ```
+- **Remote GGUF quantized (multi-shard subfolder)**  
+  ```bash
+  target/release/xinfer --d 0,1,2,3 --m unsloth/GLM-5.2-GGUF --f UD-Q2_K_XL --ui-server
+  ```
+- **Remote GGUF quantized (single file)**  
   ```bash
   target/release/xinfer --m unsloth/Qwen3-0.6B-GGUF --f Qwen3-0.6B-Q4_K_M.gguf --ui-server
   ```
@@ -65,6 +78,52 @@ Reasoning defaults to enabled when a request omits `thinking` / `enable_thinking
   ```
 - **Graph capture**: CUDA graph is auto-enabled with `cuda` feature. Use `--disable-cuda-graph` at runtime to skip graph capture.
 
+## 4.1) Multi-node tensor parallelism
+
+Distribute tensor-parallel inference across multiple machines via TCP-based NCCL bootstrap. Node 0 is the coordinator (runs scheduler + API); worker nodes run forward-only daemon loops.
+
+**Requirements:**
+- All nodes must have the same model weights available locally (same path or HuggingFace cache).
+- All nodes must be reachable via TCP on the `--master-port` (default 29500) and `--master-port + 1` (forward coordination).
+- Build with `--features cuda,nccl,flashinfer,cutlass` on all nodes.
+
+**Example: 2 nodes × 4 GPUs = 8-way tensor parallelism**
+
+```bash
+# Node 0 (master, at 192.168.1.100): runs scheduler + API server
+xinfer --m /data/DeepSeek-R1/ --d 0,1,2,3 \
+  --num-nodes 2 --node-rank 0 \
+  --master-addr 192.168.1.100 --master-port 29500 \
+  --ui-server
+
+# Node 1 (worker, at 192.168.1.101): runs forward-only daemon
+xinfer --m /data/DeepSeek-R1/ --d 0,1,2,3 \
+  --num-nodes 2 --node-rank 1 \
+  --master-addr 192.168.1.100 --master-port 29500
+```
+
+**How it works:**
+1. Node 0 generates a NCCL unique ID and distributes it to worker nodes via TCP.
+2. Each node spawns local runner subprocesses with global NCCL ranks (`node_rank × local_gpus + local_rank`).
+3. All ranks join a single global NCCL communicator for all-reduce / all-gather operations.
+4. Node 0 broadcasts forward-pass commands to worker nodes via TCP; NCCL synchronizes the actual tensor computations.
+5. Only node 0 runs the scheduler and API server; worker nodes are stateless forward engines.
+
+**CLI flags:**
+| Flag | Default | Description |
+|------|---------|-------------|
+| `--num-nodes` | 1 | Total number of nodes |
+| `--node-rank` | 0 | This node's rank (0 = master) |
+| `--master-addr` | _(required)_ | Master node's IP address |
+| `--master-port` | 29500 | TCP port for NCCL bootstrap |
+
+**NCCL environment variables** (optional tuning):
+```bash
+export NCCL_IB_DISABLE=1      # Disable InfiniBand if unavailable
+export NCCL_SOCKET_IFNAME=eth0 # Specify network interface
+export NCCL_DEBUG=INFO          # Enable NCCL debug logging
+```
+
 ## 5) PD Disaggregation (prefill/decoding split)
 - **PD server (prefill host, usually memory-rich)**  
   ```bash

package/package.json

@@ -1,7 +1,7 @@
 {
   "name": "xinfer-ai",
-  "version": "0.13.3",
-  "assetVersion": "0.13.3",
+  "version": "0.13.5",
+  "assetVersion": "0.13.5",
   "description": "xInfer — a high-performance LLM inference engine in Rust with CUDA/Metal acceleration",
   "license": "MIT",
   "homepage": "https://github.com/guoqingbao/xinfer#readme",