xinfer-ai 0.14.0 → 0.14.2

package/README.md

@@ -98,9 +98,10 @@ Add `--kvcache-dtype` to compress KV cache and extend context length:
 | QwQ-32B | Q4_K_M | 32B | **46.02** tokens/s |
 | **Qwen3-30B-A3B** | NVFP4 | **30B (MoE)** | **197.29** tokens/s (**RTX 5090**) |
 | **Qwen3-30B-A3B** | NVFP4 | **30B (MoE)** | **72.86** tokens/s (**V100, Software FP4**) |
+| **Qwen3-Next-80B** | AWQ | **80B (MoE)** | **90** tokens/s (**Hopper**) |
 | **Qwen3.5-27B** (**Multimodal**) | Q4_K_M | **27B (Dense)** | **49.33** tokens/s |
 | **Qwen3.5-27B/Qwen3.6-27B** | FP8 | **27B (Dense)** | **45** tokens/s (**Hopper**) |
-| **Qwen3.6-35B-A3B** (**Multimodal**) | FP8 | **35B (MoE)** | **110** tokens/s (**Hopper**) |
+| **Qwen3.6-35B-A3B** (**Multimodal**) | FP8 | **35B (MoE)** | **120** tokens/s (**Hopper**) |
 | **GLM4.7 Flash** | NVFP4 | **30B (MoE)** | **79** tokens/s (**Hopper, Software FP4**) |
 | **Gemma4-31B** | ISQ (BF16→Q4K) | **31B (Dense)** | **47** tokens/s (**Hopper**) |
 | **Gemma4-26B-A4B** | NVFP4 | **26B (MoE)** | **137.23** tokens/s (**RTX 5090**) |
@@ -235,6 +236,9 @@ xinfer --i --m unsloth/Qwen3.5-27B-GGUF --f Qwen3.5-27B-Q4_K_M.gguf
 # Faster GDN prefill on Hopper with slight precision loss
 SM90_LOWER_PRECISION_GDN_PREFILL=1 xinfer --m Qwen/Qwen3.5-35B-A3B-FP8
 
+# AWQ (pack-quantized)
+xinfer --m cyankiwi/Qwen3-Coder-Next-AWQ-4bit
+
 # MultiNode: GLM 5.2 (DeepSeek V3.2 architecture, FP8)
 # Master node
 xinfer --d 0,1,2,3,4,5,6,7 --m zai-org/GLM-5.2-FP8 --num-nodes 2 --node-rank 0 --master-addr 192.168.xxx.xxx
@@ -479,6 +483,7 @@ xinfer --m Qwen/Qwen3.6-27B-FP8 --ui-server --enable-tool-grammar
 | [Rust Crate](docs/rust_crate.md) | Use as a library |
 | [Add a Model](docs/add_model.md) | Port a new architecture (AI-assisted) |
 | [Test a Model](docs/test_model.md) | Validate model quality (AI-assisted) |
+| [CUDA Precision Probes](docs/precision_probes.md) | Isolate CUDA kernel precision problems |
 
 **Using Agents under xInfer backend:** [xbot](docs/xbot.md) · [OpenCode](docs/opencode.md) · [Kilo Code](docs/kilocode.md) · [Claude Code](docs/claude_code.md) · [Goose](docs/goose.md)
 

package/docs/precision_probes.md

@@ -0,0 +1,66 @@
+# CUDA Precision Probe Suite
+
+The precision suite is a kernel-level differential test harness. Use it to isolate a CUDA precision problem before attributing the symptom to a full model or sampler.
+
+## Run the complete suite
+
+From the xInfer repository root:
+
+```bash
+./tests/run_precision_suite.sh
+```
+
+The runner:
+
+1. Detects every visible GPU's compute capability with `nvidia-smi` and uses the lowest capability for a safe multi-GPU build.
+2. Builds all Rust probes once with the release CUDA configuration.
+3. Runs each Rust probe followed by its paired Python comparator.
+
+The probes cover Candle CUDA/PTX operations, ISQ/GGUF Q4_K and Q6_K, attention.rs kernels, FP8, GDN, paged FlashInfer attention, and the NVFP4 paths supported by the detected GPU. Python comparisons use independent PyTorch references and official Python FlashInfer when available. vLLM and sglang comparisons are attempted when those packages are installed.
+
+The suite prints a retained result directory such as:
+
+```text
+/tmp/xinfer_precision_suite_sm120_<pid>
+```
+
+Read the first `FAIL` line and the corresponding stage's `[FAIL]` marker. The failing stage identifies the subsystem; the operation name identifies the kernel family or operation.
+
+## Rerun one comparator
+
+Probe binaries and their output files are retained, so a Python comparator can be rerun without rebuilding or executing the GPU probe again:
+
+```bash
+python3 tests/probes/compare_candle_probe.py \
+  /tmp/xinfer_precision_suite_sm120_<pid>/candle.bin
+
+python3 tests/probes/compare_flashinfer_probe.py \
+  /tmp/xinfer_precision_suite_sm120_<pid>/flashinfer
+```
+
+Use the corresponding comparator in `tests/probes/` for the failed stage:
+
+| Stage | Comparator |
+|---|---|
+| Candle and Q4_K/Q6_K | `compare_candle_probe.py` |
+| attention.rs common kernels | `compare_attention_misc_probe.py` |
+| FP8 | `compare_fp8_probe.py` |
+| GDN | `compare_gdn_probe.py` |
+| FlashInfer attention | `compare_flashinfer_probe.py` |
+| NVFP4 | `compare_nvfp4_probe.py` |
+
+## Optional GDN audit
+
+The SM90 persistent FlashInfer GQA path is opt-in and excluded by default. Audit it explicitly with:
+
+```bash
+./tests/run_precision_suite.sh --include-optin-gdn
+```
+
+This flag is only for auditing that optional path; it does not enable or change the production routing.
+
+## Interpreting SM capability results
+
+For SM70 and SM75, the runner automatically builds the legacy CUDA path with only `cuda,nccl`. FlashInfer and CUTLASS-dependent attention and hardware-NVFP4 cases are skipped, but the software FP8 and software NVFP4 GEMM/MoE/helper cases still run and are compared with the independent PyTorch golden. Since attention.rs disables BF16 CUDA kernels below SM80, the legacy software probes use F16 there. The CUDA toolkit must also provide an `nvcc` that supports `compute_70`/`compute_75`; CUDA 12.6 is the recommended legacy-toolkit choice. CUDA 13 `nvcc` rejects these targets before Rust compilation begins.
+
+On SM90, Blackwell-only NVFP4 hardware helpers are reported as `SKIP` because they cannot execute on that GPU. Run the suite on SM100/SM120 hardware to exercise hardware NVFP4 dense, GEMM, and MoE paths. Missing optional Python packages are reported explicitly; install versions matching the CUDA and PyTorch environment when those golden comparisons are required.

package/package.json

@@ -1,7 +1,7 @@
 {
   "name": "xinfer-ai",
-  "version": "0.14.0",
-  "assetVersion": "0.14.0",
+  "version": "0.14.2",
+  "assetVersion": "0.14.2",
   "description": "xInfer — a high-performance LLM inference engine in Rust with CUDA/Metal acceleration",
   "license": "MIT",
   "homepage": "https://github.com/guoqingbao/xinfer#readme",