xinfer-ai 0.12.6 → 0.12.8

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (3) hide show
  1. package/README.md +15 -15
  2. package/docs/performance.md +15 -15
  3. package/package.json +2 -2
package/README.md CHANGED
@@ -11,7 +11,7 @@
11
11
  | | Feature | Details |
12
12
  |---|---|---|
13
13
  | **0️⃣** | Zero Python dependencies | Pure Rust backend — no PyTorch, no CUDA Python bindings |
14
- | **⚡** | Fast | Native Flash Attention, FlashInfer, CUDA Graphs, continuous batching, prefix caching, PD disaggregation. Up to **175 tok/s** decode for `30B+` models on consumer GPUs |
14
+ | **⚡** | Fast | Native Flash Attention, FlashInfer, CUDA Graphs, continuous batching, prefix caching, PD disaggregation. Up to **181 tok/s** decode for `30B+` models on consumer GPUs |
15
15
  | **🪶** | Tiny footprint | Core scheduling + attention logic in **< 5 000 lines** of Rust |
16
16
  | **🌍** | Cross-platform | CUDA (Linux/Windows), Metal (macOS). Same binary, same API |
17
17
  | **🏭** | Production-ready | OpenAI/Anthropic-compatible APIs, built-in ChatGPT-style Web UI, MCP tool calling, structured outputs, embedding + tokenizer endpoints |
@@ -79,21 +79,21 @@ Add `--kvcache-dtype` to compress KV cache and extend context length:
79
79
 
80
80
  | Model | Format | Size | Decoding Speed |
81
81
  |---|---|---|---|
82
- | Ministral-3-3B (**Multimodal**) | ISQ (BF16→Q4K) | 3B | **171.92** tokens/s |
83
- | Qwen3-VL-8B-Instruct (**Multimodal**) | Q8_0 | 8B | **105.31** tokens/s |
84
- | Llama-3.1-8B | ISQ (BF16→Q4K) | 8B | **120.74** tokens/s |
85
- | DeepSeek-R1-0528-Qwen3-8B | Q4_K_M | 8B | **124.87** tokens/s |
86
- | GLM-4-9B-0414 | Q4_K_M | 9B | **70.38** tokens/s |
87
- | QwQ-32B | Q4_K_M | 32B | **41.36** tokens/s |
88
- | **Qwen3-30B-A3B** | NVFP4 | **30B (MoE)** | **175.30** tokens/s (**RTX 5090**) |
89
- | **Qwen3-30B-A3B** | NVFP4 | **30B (MoE)** | **67.10** tokens/s (**V100, Software FP4**) |
90
- | **Qwen3.5-27B** (**Multimodal**) | Q4_K_M | **27B (Dense)** | **45.20** tokens/s |
91
- | **Qwen3.5-27B/Qwen3.6-27B** | FP8 | **27B (Dense)** | **42** tokens/s (**Hopper**) |
92
- | **Qwen3.6-35B-A3B** (**Multimodal**) | FP8 | **35B (MoE)** | **102** tokens/s (**Hopper**) |
82
+ | Ministral-3-3B (**Multimodal**) | ISQ (BF16→Q4K) | 3B | **193.67** tokens/s |
83
+ | Qwen3-VL-8B-Instruct (**Multimodal**) | Q8_0 | 8B | **112.51** tokens/s |
84
+ | Llama-3.1-8B | ISQ (BF16→Q4K) | 8B | **133.10** tokens/s |
85
+ | DeepSeek-R1-0528-Qwen3-8B | Q4_K_M | 8B | **139.25** tokens/s |
86
+ | GLM-4-9B-0414 | Q4_K_M | 9B | **77.48** tokens/s |
87
+ | QwQ-32B | Q4_K_M | 32B | **46.02** tokens/s |
88
+ | **Qwen3-30B-A3B** | NVFP4 | **30B (MoE)** | **181.59** tokens/s (**RTX 5090**) |
89
+ | **Qwen3-30B-A3B** | NVFP4 | **30B (MoE)** | **72.86** tokens/s (**V100, Software FP4**) |
90
+ | **Qwen3.5-27B** (**Multimodal**) | Q4_K_M | **27B (Dense)** | **49.33** tokens/s |
91
+ | **Qwen3.5-27B/Qwen3.6-27B** | FP8 | **27B (Dense)** | **45** tokens/s (**Hopper**) |
92
+ | **Qwen3.6-35B-A3B** (**Multimodal**) | FP8 | **35B (MoE)** | **110** tokens/s (**Hopper**) |
93
93
  | **GLM4.7 Flash** | NVFP4 | **30B (MoE)** | **79** tokens/s (**Hopper, Software FP4**) |
94
- | **Gemma4-31B** | ISQ (BF16→Q4K) | **31B (Dense)** | **41** tokens/s (**Hopper**) |
95
- | **Gemma4-26B-A4B** | NVFP4 | **26B (MoE)** | **131** tokens/s (**RTX 5090**) |
96
- | **MiniMax-M2.5** | NVFP4 | **229B (MoE)** | **62** tokens/s (**Hopper, Software FP4, TP=2**) |
94
+ | **Gemma4-31B** | ISQ (BF16→Q4K) | **31B (Dense)** | **47** tokens/s (**Hopper**) |
95
+ | **Gemma4-26B-A4B** | NVFP4 | **26B (MoE)** | **137.23** tokens/s (**RTX 5090**) |
96
+ | **MiniMax-M2.5** | NVFP4 | **229B (MoE)** | **64.50** tokens/s (**Hopper, Software FP4, TP=2**) |
97
97
 
98
98
  <details>
99
99
  <summary><b>Apple Silicon (M4)</b></summary>
package/docs/performance.md CHANGED
@@ -10,21 +10,21 @@ This document contains detailed performance benchmarks for xInfer across differe
10
10
 
11
11
  | Model | Format | Size | Hardware | Decoding Speed |
12
12
  |-------|--------|------|----------|----------------|
13
- | Qwen3-30B-A3B | NVFP4 | 30B MoE | RTX 5090 (SM120) | **175.30** tokens/s |
14
- | Gemma4-26B-A4B | NVFP4 | 26B MoE | RTX 5090 (SM120) | **131** tokens/s |
15
- | Ministral-3-3B (Multimodal) | ISQ (BF16→Q4K) | 3B | A100 (SM80) | **171.92** tokens/s |
16
- | DeepSeek-R1-0528-Qwen3-8B | Q4_K_M | 8B | A100 (SM80) | **124.87** tokens/s |
17
- | Llama-3.1-8B | ISQ (BF16→Q4K) | 8B | A100 (SM80) | **120.74** tokens/s |
18
- | Qwen3-VL-8B-Instruct (Multimodal) | Q8_0 | 8B | A100 (SM80) | **105.31** tokens/s |
19
- | Qwen3.6-35B-A3B (Multimodal) | FP8 | 35B MoE | H800 (SM90) | **102** tokens/s |
20
- | GLM4.7 Flash | NVFP4 | 30B MoE | H800 (SM90) | **79** tokens/s |
21
- | Qwen3-30B-A3B | NVFP4 | 30B MoE | V100 (SM70) | **67.10** tokens/s |
22
- | GLM-4-9B-0414 | Q4_K_M | 9B | A100 (SM80) | **70.38** tokens/s |
23
- | MiniMax-M2.5 | NVFP4 | 229B MoE | H800 ×2 (SM90) | **62** tokens/s |
24
- | Qwen3.5-27B (Multimodal) | Q4_K_M | 27B Dense | A100 (SM80) | **45.20** tokens/s |
25
- | Qwen3.5-27B/Qwen3.6-27B | FP8 | 27B Dense | H800 (SM90) | **42** tokens/s |
26
- | QwQ-32B | Q4_K_M | 32B | A100 (SM80) | **41.36** tokens/s |
27
- | Gemma4-31B | ISQ (BF16→Q4K) | 31B Dense | H800 (SM90) | **41** tokens/s |
13
+ | Ministral-3-3B (**Multimodal**) | ISQ (BF16→Q4K) | 3B | A100| **193.67** tokens/s |
14
+ | Qwen3-VL-8B-Instruct (**Multimodal**) | Q8_0 | 8B | A100| **112.51** tokens/s |
15
+ | Llama-3.1-8B | ISQ (BF16→Q4K) | 8B | A100| **133.10** tokens/s |
16
+ | DeepSeek-R1-0528-Qwen3-8B | Q4_K_M | 8B | A100| **139.25** tokens/s |
17
+ | GLM-4-9B-0414 | Q4_K_M | 9B | A100| **77.48** tokens/s |
18
+ | QwQ-32B | Q4_K_M | 32B | A100| **46.02** tokens/s |
19
+ | **Qwen3-30B-A3B** | NVFP4 | **30B (MoE)** | RTX 5090 | **181.59** tokens/s|
20
+ | **Qwen3-30B-A3B** | NVFP4 | **30B (MoE)** | V100 | **72.86** tokens/s (**Software FP4**) |
21
+ | **Qwen3.5-27B** (**Multimodal**) | Q4_K_M | **27B (Dense)** | Hopper | **49.33** tokens/s |
22
+ | **Qwen3.5-27B/Qwen3.6-27B** | FP8 | **27B (Dense)** | Hopper | **45** tokens/s|
23
+ | **Qwen3.6-35B-A3B** (**Multimodal**) | FP8 | **35B (MoE)** | Hopper | **110** tokens/s |
24
+ | **GLM4.7 Flash** | NVFP4 | **30B (MoE)** | Hopper | **79** tokens/s (**Software FP4**) |
25
+ | **Gemma4-31B** | ISQ (BF16→Q4K) | **31B (Dense)** | Hopper| **47** tokens/s |
26
+ | **Gemma4-26B-A4B** | NVFP4 | **26B (MoE)** | RTX 5090 | **137.23** tokens/s|
27
+ | **MiniMax-M2.5** | NVFP4 | **229B (MoE)** | Hopper | **64.50** tokens/s (**Software FP4, TP=2**) |
28
28
 
29
29
  ### V100 + NVFP4 + TurboQuant (First-Ever)
30
30
 
package/package.json CHANGED
@@ -1,7 +1,7 @@
1
1
  {
2
2
  "name": "xinfer-ai",
3
- "version": "0.12.6",
4
- "assetVersion": "0.12.6",
3
+ "version": "0.12.8",
4
+ "assetVersion": "0.12.8",
5
5
  "description": "xInfer — a high-performance LLM inference engine in Rust with CUDA/Metal acceleration",
6
6
  "license": "MIT",
7
7
  "homepage": "https://github.com/guoqingbao/xinfer#readme",