xinfer-ai 0.14.4 → 0.14.5

package/README.md

@@ -61,11 +61,23 @@ xinfer --m /home/Qwen3.6-35B-A3B --d 0,1 --ui-server
 python3 -m xinfer.server --m Qwen/Qwen3.6-27B-FP8 --kvcache-dtype turbo4 --ui-server
 ```
 
-**MTP**
+**Speculative decoding (MTP & DFlash2)**
+
+Built-in MTP (model must include MTP heads):
+
 ```bash
-xinfer --w /home/Qwen3.6-35B-A3B --d 0,1 --ui-server --mtp 2
+xinfer --m Qwen/Qwen3.5-35B-A3B --d 0,1 --ui-server --num-speculative-tokens 3
 ```
 
+External DFlash2 draft model:
+
+```bash
+xinfer --m Qwen/Qwen3.8-... --d 0,1 --ui-server \
+  --draft-model <dflash2-draft-id-or-path> --num-speculative-tokens 7
+```
+
+See [docs/speculative_decoding.md](docs/speculative_decoding.md) for details.
+
 > **Tip:** Open `http://IP:8001` for the built-in chat UI, or use `http://IP:8000/v1/` as your API `Base URL`.
 
 ---
@@ -134,6 +146,7 @@ Add `--kvcache-dtype` to compress KV cache and extend context length:
 * ✅ Qwen2/Qwen3 MoE
 * ✅ Qwen3 Next
 * ✅ Qwen3.5/3.6/3.8 Dense/MoE (27B, 35B, 122B, 397B, Multimodal model)
+* ✅ Qwen3.8-Next/Qwen4
 * ✅ Mistral v1, v2
 * ✅ Mistral-3-VL Reasoning (3B, 8B, 14B, Multimodal model)
 * ✅ GLM4 (0414)
@@ -529,7 +542,8 @@ xinfer --m Qwen/Qwen3.6-27B-FP8 --ui-server --enable-tool-grammar
 | `--frequency-penalty` | Penalize frequent tokens (−2 to 2) |
 | `--mcp-config` | MCP servers JSON config |
 | `--mcp-command` / `--mcp-args` | Single MCP server command + args |
-| `--mtp`| Multi-token prediction, usage `--mtp 2` for two-token prediction per forward pass |
+| `--num-speculative-tokens` | Speculative draft tokens per decode step. Enables **built-in MTP** on models with MTP heads (e.g. Qwen3.5). Example: `--num-speculative-tokens 3` |
+| `--draft-model` | External **DFlash2** draft model (HuggingFace id or local directory). When set, uses DFlash2 instead of built-in MTP. |
 
 ### Environment Variables
 
@@ -591,7 +605,7 @@ SM90_LOWER_PRECISION_GDN_PREFILL=1 xinfer --m Qwen/Qwen3.5-35B-A3B-FP8 --ui-serv
 * [x] **MXFP4/NVFP4 Model Support**
 * [x] **Support Turboquant (4-bit, 3-bit) KvCache**
 * [ ] TentorRT-LLM
-* [x] **Multi-token Prediciton (MTP)**
+* [x] **Multi-token Prediction (MTP) & DFlash2 speculative decoding**
 
 ---
 

package/docs/get_started.md

@@ -67,10 +67,27 @@ cargo build --release --features metal
   target/release/xinfer --m Qwen/Qwen3-VL-8B-Instruct --ui-server
   ```
 
-Common runtime knobs: `--max-model-len`, `--max-num-seqs`, `--kv-fraction` (CUDA KV share), `--cpu-mem-fold` (CPU swap ratio), `--port`, `--kvcache-dtype` (fp8/turbo8/turbo4/turbo3), `--prefix-cache`, `--prefix-cache-max-tokens`, `--ui-server`, `--batch` (perf test).
+Common runtime knobs: `--max-model-len`, `--max-num-seqs`, `--kv-fraction` (CUDA KV share), `--cpu-mem-fold` (CPU swap ratio), `--port`, `--kvcache-dtype` (fp8/turbo8/turbo4/turbo3), `--prefix-cache`, `--prefix-cache-max-tokens`, `--ui-server`, `--batch` (perf test), `--num-speculative-tokens` (built-in MTP), `--draft-model` (DFlash2).
 
 Reasoning defaults to enabled when a request omits `thinking` / `enable_thinking`. Use `--disable-reasoning` on the Rust CLI to make the default be disabled instead; explicit request values still override the server default.
 
+### Speculative decoding (MTP & DFlash2)
+
+Speed up decode with speculative token drafting + target verification. See [speculative_decoding.md](./speculative_decoding.md) for full details.
+
+- **Built-in MTP** (Qwen3.5 / Qwen3.6 with MTP weights): `--num-speculative-tokens N`
+  ```bash
+  target/release/xinfer --m Qwen/Qwen3.5-35B-A3B --d 0,1 \
+    --num-speculative-tokens 3 --ui-server
+  ```
+- **DFlash2** (external draft checkpoint): `--draft-model <hf_id_or_local_path>` plus optional `--num-speculative-tokens`
+  ```bash
+  target/release/xinfer --m Qwen/Qwen3.8-... --d 0,1 \
+    --draft-model <dflash2-draft> --num-speculative-tokens 7 --ui-server
+  ```
+
+If `--draft-model` is set, DFlash2 is used instead of built-in MTP. Build with `cuda,nccl,flashinfer,cutlass` for best performance.
+
 ## 4) Multi-rank (single node)
 - **NCCL multi-GPU**  
   ```bash

package/docs/speculative_decoding.md

@@ -0,0 +1,129 @@
+# Speculative Decoding (MTP & DFlash2)
+
+xInfer speeds up autoregressive decode with **speculative decoding**: a small set of candidate tokens is proposed each step, then verified by the target model in a single batched forward pass. Accepted drafts advance the sequence without extra per-token decode rounds.
+
+Two modes are supported:
+
+| Mode | When to use | CLI |
+|------|-------------|-----|
+| **Built-in MTP** | Target model ships MTP prediction heads (e.g. Qwen3.5 / Qwen3.6 with MTP weights) | `--num-speculative-tokens N` |
+| **DFlash2** | External DFlash2 draft model (separate safetensors checkpoint) | `--draft-model <id_or_path>` |
+
+**Rule:** If `--draft-model` is set, xInfer uses **DFlash2** (DFlash1 checkpoints are not supported). Otherwise, `--num-speculative-tokens` enables **built-in MTP** when the target model has MTP layers.
+
+---
+
+## Built-in MTP
+
+MTP (Multi-Token Prediction) uses lightweight heads bundled in the target checkpoint. Each decode step:
+
+1. Target model produces one **anchor** token and a hidden state.
+2. MTP head drafts `N` tokens autoregressively (no extra KV growth).
+3. Target model verifies `[anchor, draft₀, …, draftₙ₋₁]` in one forward pass.
+4. Matching prefix is accepted; GDN/Mamba state is rolled back on partial rejection (Qwen3.5 hybrid models).
+
+### CLI
+
+```bash
+# Qwen3.5 35B with 3 speculative tokens per step
+xinfer --m Qwen/Qwen3.5-35B-A3B --d 0,1 \
+  --num-speculative-tokens 3 --ui-server
+```
+
+Typical values: **3–7** draft tokens. Higher values increase acceptance variance and KV verify width.
+
+### Requirements
+
+- Target weights must include MTP layers (`mtp.*` or GGUF `nextn.*` tensors).
+- Model types with MTP support today: **Qwen3.5**, **Qwen3.5 MoE**, **Qwen3-VL** (Qwen3.5 text backbone).
+- Recommended build: `./build.sh --release --features cuda,nccl,flashinfer,cutlass`
+
+---
+
+## DFlash2
+
+[DFlash2](https://github.com/z-lab/dflash) uses a **separate draft model** that consumes projected intermediate hidden states from selected target layers. Draft tokens are chosen with a top-k candidate lattice (`selector_top_k` in the draft config).
+
+### CLI
+
+```bash
+# Qwen3.8 target + DFlash2 draft (HuggingFace id or local path)
+xinfer --m Qwen/Qwen3.8-... --d 0,1 \
+  --draft-model <your-dflash2-draft-repo-or-path> \
+  --num-speculative-tokens 7 --ui-server
+```
+
+`--draft-model` accepts either:
+
+- A **HuggingFace model id** (downloads `config.json` + safetensors), or
+- A **local directory** containing `config.json` and `model.safetensors` (or sharded index).
+
+`--num-speculative-tokens` is the number of draft tokens proposed per step (excluding the anchor). If omitted, xInfer uses the draft model config (`block_size - 1`).
+
+Draft weights must be **safetensors** (GGUF draft models are not supported). The checkpoint must be **DFlash2** (`architectures` contains `DFlash2` or `dflash_config.selector_top_k` is set).
+
+### Requirements
+
+- Target: **Qwen3.5**, **Qwen3.5 MoE**, **Qwen3-VL**, or **Qwen3.8** (dense) today.
+- Draft: matching-family **DFlash2** checkpoint.
+- Recommended build: `cuda,nccl,flashinfer,cutlass` (CUDA graphs optional; disable with `--disable-cuda-graph` for debugging).
+
+---
+
+## Python API
+
+```python
+from xinfer import Engine, EngineConfig
+
+# Built-in MTP
+cfg = EngineConfig(
+    model_id="Qwen/Qwen3.5-35B-A3B",
+    num_speculative_tokens=3,
+)
+
+# DFlash2 (draft_model enables DFlash2 instead of MTP)
+cfg = EngineConfig(
+    model_id="Qwen/Qwen3.8-...",
+    draft_model="<dflash2-draft-id-or-path>",
+    num_speculative_tokens=7,
+)
+
+engine = Engine(cfg, "bf16")
+```
+
+---
+
+## Flags reference
+
+| Flag | Description |
+|------|-------------|
+| `--num-speculative-tokens N` | Draft tokens per decode step. Enables **MTP** when the target has MTP heads and `--draft-model` is unset. With `--draft-model`, sets DFlash2 draft width (optional; defaults from draft config). |
+| `--draft-model <id_or_path>` | External **DFlash2** draft model. Enables DFlash2 and disables built-in MTP. |
+
+**Removed:** `--mtp` (use `--num-speculative-tokens`), `--draft-model-id` / `--draft-model-path` (use `--draft-model`).
+
+---
+
+## Tips & limitations
+
+- **Throughput:** Speedup depends on draft acceptance rate and model size. Monitor logs for `MTP Stats` / `DFlash2 Stats` acceptance summaries.
+- **Memory:** Verify passes append up to `N+1` tokens per step; ensure KV budget via `--kv-fraction` and `--max-num-seqs`.
+- **Hybrid models:** Qwen3.5 GDN layers require state rollback on rejected drafts; this is handled automatically.
+- **Batching:** Speculative decode is optimized for `batch_size=1` decode steps; larger batches use a batched DFlash2 path.
+- **PD disaggregation:** Do not combine speculative decode with PD client/server modes on hybrid Mamba models.
+
+---
+
+## Quick test
+
+```bash
+# 1. Health check
+curl -s http://localhost:8000/v1/models
+
+# 2. Short completion (MTP or DFlash2 server)
+curl -s http://localhost:8000/v1/chat/completions \
+  -H "Content-Type: application/json" \
+  -d '{"model":"Qwen/Qwen3.5-35B-A3B","messages":[{"role":"user","content":"Hello"}],"max_tokens":64}'
+```
+
+See also [test_model.md](./test_model.md) for full model validation workflows.

package/package.json

@@ -1,7 +1,7 @@
 {
   "name": "xinfer-ai",
-  "version": "0.14.4",
-  "assetVersion": "0.14.4",
+  "version": "0.14.5",
+  "assetVersion": "0.14.5",
   "description": "xInfer — a high-performance LLM inference engine in Rust with CUDA/Metal acceleration",
   "license": "MIT",
   "homepage": "https://github.com/guoqingbao/xinfer#readme",