xinfer-ai 0.14.1 → 0.14.3

package/README.md

@@ -141,12 +141,15 @@ Add `--kvcache-dtype` to compress KV cache and extend context length:
 * ✅ GLM4.7 Flash
 * ✅ GLM 5.2 (DeepSeek V3.2 DSA architecture)
 * ✅ DeepSeek V3/R1/V3.2
+* ✅ **DeepSeek V4** (support 2-bit ISQ via `--isq w2`)
 * ✅ Phi3 / Phi4 (Phi-3, Phi-4, Phi-4-mini, etc.)
 * ✅ Gemma3/**Gemma4** (Multimodal model)
 * ✅ Qwen3-VL (Dense, Multimodal model)
 * ✅ MiroThinker-v1.5 (30B, 235B)
 
-**Formats:** Safetensors (BF16, `FP8-blockwise`, GPTQ, AWQ, MXFP4, `NVFP4`) | GGUF (all quant types) | `ISQ` (on-the-fly quantization)
+**Formats:** Safetensors (BF16, `FP8-blockwise`, GPTQ, AWQ, MXFP4, `NVFP4`) | GGUF (all quant types) | `ISQ` (on-the-fly quantization, including **2-bit W2** for DeepSeek V4 MoE)
+
+> **DeepSeek V4 Flash + 2-bit ISQ:** `--isq w2` re-packs MXFP4 routed experts to 2-bit on load. With 2× GPU tensor parallel (`--d 0,1`), model weights use about **46 GB per GPU** (**46 × 2 GB** total).
 
 ---
 
@@ -209,6 +212,9 @@ xinfer --d 0,1 --m Qwen/Qwen3-30B-A3B-Instruct-2507 --kvcache-dtype fp8
 # ISQ on-the-fly quantization
 xinfer --m Qwen/Qwen3.6-35B-A3B --isq q4k
 
+# DeepSeek V4 Flash (2-bit ISQ MoE, ~46 GB weights/GPU on 2× TP)
+xinfer --d 0,1 --m /path/DeepSeek-V4-Flash --isq w2 --kv-fraction 0.7 --ui-server
+
 # NVFP4 model
 xinfer --m unsloth/Qwen3.6-27B-NVFP4
 
@@ -258,6 +264,9 @@ xinfer --m Qwen/Qwen3.6-35B-A3B --isq q4k --kvcache-dtype turbo4
 
 # Metal ISQ
 xinfer --m /path/Qwen3-4B --isq q6k
+
+# DeepSeek V4 Flash: 2-bit W2 ISQ for routed MoE experts (~46 GB weights/GPU, 2× TP)
+xinfer --d 0,1 --m /path/DeepSeek-V4-Flash --isq w2 --kv-fraction 0.7
 ```
 
 </details>
@@ -483,6 +492,7 @@ xinfer --m Qwen/Qwen3.6-27B-FP8 --ui-server --enable-tool-grammar
 | [Rust Crate](docs/rust_crate.md) | Use as a library |
 | [Add a Model](docs/add_model.md) | Port a new architecture (AI-assisted) |
 | [Test a Model](docs/test_model.md) | Validate model quality (AI-assisted) |
+| [CUDA Precision Probes](docs/precision_probes.md) | Isolate CUDA kernel precision problems |
 
 **Using Agents under xInfer backend:** [xbot](docs/xbot.md) · [OpenCode](docs/opencode.md) · [Kilo Code](docs/kilocode.md) · [Claude Code](docs/claude_code.md) · [Goose](docs/goose.md)
 
@@ -499,7 +509,7 @@ xinfer --m Qwen/Qwen3.6-27B-FP8 --ui-server --enable-tool-grammar
 | `--ui-server` | API server + built-in ChatGPT-style web UI |
 | `--server` | API server. Bare: `0.0.0.0:8000`; or `host[:port]`, `file:///path`, `socket:///path`, `unix:///path` |
 | `--i` | Interactive CLI chat |
-| `--isq` | On-the-fly quantization: `q2k`, `q3k`, `q4k`, `q5k`, `q6k`, `q8_0` |
+| `--isq` | On-the-fly quantization: `q2k`, `q3k`, `q4k`, `q5k`, `q6k`, `q8_0`, `w2` / `moe_w2` |
 | `--kvcache-dtype` | KV cache quantization: `fp8`, `turbo8`, `turbo4`, `turbo3` |
 | `--max-num-seqs` | Max concurrent requests (default: 32, macOS: 8) |
 | `--max-tokens` | Max tokens per response (default: 16384) |
@@ -526,6 +536,7 @@ xinfer --m Qwen/Qwen3.6-27B-FP8 --ui-server --enable-tool-grammar
 | Variable | Description |
 |---|---|
 | `XINFER_NVFP4_FORCE_LUT=1` | Force software NVFP4 decode to use the LUT-based dequantization path (higher precision) instead of hardware FP4 intrinsics on Blackwell (SM100+). Useful when decode precision matters more than peak throughput. |
+| `XINFER_ENABLE_FLASHMLA` | Enables DeepSeek V4 sparse MLA acceleration when set to `1` or `true`. It is disabled by default (unset), so the precision-preserving custom BF16 sparse-attention kernel is used. |
 | `XINFER_SSE_BUFFER_SIZE=256` | Size of the bounded SSE streaming buffer per client connection (default: 256). Increase for slow network proxies or high-throughput models. |
 | `SM90_LOWER_PRECISION_GDN_PREFILL=1` | Enable the FlashInfer SM90 persistent kernel for GatedDeltaNet (GDN) prefill on Hopper GPUs (SM90). Delivers faster prefill speedup for Qwen3.5/3.6, with a slight precision trade-off. |
 
@@ -563,6 +574,7 @@ SM90_LOWER_PRECISION_GDN_PREFILL=1 xinfer --m Qwen/Qwen3.5-35B-A3B-FP8 --ui-serv
 * [x] TurboQuant KV Cache (2-4 bit compression with WHT rotation)
 * [x] FP8 Models (CUDA: MoE, Dense; Metal: Dense)
 * [x] GLM 5.2 (DeepSeek V3.2 + DSA)
+* [x] DeepSeek V4 (Flash) + 2-bit W2 ISQ MoE (~46 GB weights/GPU × 2)
 * [ ] Additional model support (Kimi K2, etc.)
 * [x] CPU KV Cache Offloading
 * [x] Prefill-decode Disaggregation (CUDA)

package/docs/precision_probes.md

@@ -0,0 +1,66 @@
+# CUDA Precision Probe Suite
+
+The precision suite is a kernel-level differential test harness. Use it to isolate a CUDA precision problem before attributing the symptom to a full model or sampler.
+
+## Run the complete suite
+
+From the xInfer repository root:
+
+```bash
+./tests/run_precision_suite.sh
+```
+
+The runner:
+
+1. Detects every visible GPU's compute capability with `nvidia-smi` and uses the lowest capability for a safe multi-GPU build.
+2. Builds all Rust probes once with the release CUDA configuration.
+3. Runs each Rust probe followed by its paired Python comparator.
+
+The probes cover Candle CUDA/PTX operations, ISQ/GGUF Q4_K and Q6_K, attention.rs kernels, FP8, GDN, paged FlashInfer attention, and the NVFP4 paths supported by the detected GPU. Python comparisons use independent PyTorch references and official Python FlashInfer when available. vLLM and sglang comparisons are attempted when those packages are installed.
+
+The suite prints a retained result directory such as:
+
+```text
+/tmp/xinfer_precision_suite_sm120_<pid>
+```
+
+Read the first `FAIL` line and the corresponding stage's `[FAIL]` marker. The failing stage identifies the subsystem; the operation name identifies the kernel family or operation.
+
+## Rerun one comparator
+
+Probe binaries and their output files are retained, so a Python comparator can be rerun without rebuilding or executing the GPU probe again:
+
+```bash
+python3 tests/probes/compare_candle_probe.py \
+  /tmp/xinfer_precision_suite_sm120_<pid>/candle.bin
+
+python3 tests/probes/compare_flashinfer_probe.py \
+  /tmp/xinfer_precision_suite_sm120_<pid>/flashinfer
+```
+
+Use the corresponding comparator in `tests/probes/` for the failed stage:
+
+| Stage | Comparator |
+|---|---|
+| Candle and Q4_K/Q6_K | `compare_candle_probe.py` |
+| attention.rs common kernels | `compare_attention_misc_probe.py` |
+| FP8 | `compare_fp8_probe.py` |
+| GDN | `compare_gdn_probe.py` |
+| FlashInfer attention | `compare_flashinfer_probe.py` |
+| NVFP4 | `compare_nvfp4_probe.py` |
+
+## Optional GDN audit
+
+The SM90 persistent FlashInfer GQA path is opt-in and excluded by default. Audit it explicitly with:
+
+```bash
+./tests/run_precision_suite.sh --include-optin-gdn
+```
+
+This flag is only for auditing that optional path; it does not enable or change the production routing.
+
+## Interpreting SM capability results
+
+For SM70 and SM75, the runner automatically builds the legacy CUDA path with only `cuda,nccl`. FlashInfer and CUTLASS-dependent attention and hardware-NVFP4 cases are skipped, but the software FP8 and software NVFP4 GEMM/MoE/helper cases still run and are compared with the independent PyTorch golden. Since attention.rs disables BF16 CUDA kernels below SM80, the legacy software probes use F16 there. The CUDA toolkit must also provide an `nvcc` that supports `compute_70`/`compute_75`; CUDA 12.6 is the recommended legacy-toolkit choice. CUDA 13 `nvcc` rejects these targets before Rust compilation begins.
+
+On SM90, Blackwell-only NVFP4 hardware helpers are reported as `SKIP` because they cannot execute on that GPU. Run the suite on SM100/SM120 hardware to exercise hardware NVFP4 dense, GEMM, and MoE paths. Missing optional Python packages are reported explicitly; install versions matching the CUDA and PyTorch environment when those golden comparisons are required.

package/package.json

@@ -1,7 +1,7 @@
 {
   "name": "xinfer-ai",
-  "version": "0.14.1",
-  "assetVersion": "0.14.1",
+  "version": "0.14.3",
+  "assetVersion": "0.14.3",
   "description": "xInfer — a high-performance LLM inference engine in Rust with CUDA/Metal acceleration",
   "license": "MIT",
   "homepage": "https://github.com/guoqingbao/xinfer#readme",