xinfer-ai 0.13.8 → 0.14.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +5 -1
- package/package.json +2 -2
package/README.md
CHANGED
|
@@ -98,9 +98,10 @@ Add `--kvcache-dtype` to compress KV cache and extend context length:
|
|
|
98
98
|
| QwQ-32B | Q4_K_M | 32B | **46.02** tokens/s |
|
|
99
99
|
| **Qwen3-30B-A3B** | NVFP4 | **30B (MoE)** | **197.29** tokens/s (**RTX 5090**) |
|
|
100
100
|
| **Qwen3-30B-A3B** | NVFP4 | **30B (MoE)** | **72.86** tokens/s (**V100, Software FP4**) |
|
|
101
|
+
| **Qwen3-Next-80B** | AWQ | **80B (MoE)** | **90** tokens/s (**Hopper**) |
|
|
101
102
|
| **Qwen3.5-27B** (**Multimodal**) | Q4_K_M | **27B (Dense)** | **49.33** tokens/s |
|
|
102
103
|
| **Qwen3.5-27B/Qwen3.6-27B** | FP8 | **27B (Dense)** | **45** tokens/s (**Hopper**) |
|
|
103
|
-
| **Qwen3.6-35B-A3B** (**Multimodal**) | FP8 | **35B (MoE)** | **
|
|
104
|
+
| **Qwen3.6-35B-A3B** (**Multimodal**) | FP8 | **35B (MoE)** | **120** tokens/s (**Hopper**) |
|
|
104
105
|
| **GLM4.7 Flash** | NVFP4 | **30B (MoE)** | **79** tokens/s (**Hopper, Software FP4**) |
|
|
105
106
|
| **Gemma4-31B** | ISQ (BF16→Q4K) | **31B (Dense)** | **47** tokens/s (**Hopper**) |
|
|
106
107
|
| **Gemma4-26B-A4B** | NVFP4 | **26B (MoE)** | **137.23** tokens/s (**RTX 5090**) |
|
|
@@ -235,6 +236,9 @@ xinfer --i --m unsloth/Qwen3.5-27B-GGUF --f Qwen3.5-27B-Q4_K_M.gguf
|
|
|
235
236
|
# Faster GDN prefill on Hopper with slight precision loss
|
|
236
237
|
SM90_LOWER_PRECISION_GDN_PREFILL=1 xinfer --m Qwen/Qwen3.5-35B-A3B-FP8
|
|
237
238
|
|
|
239
|
+
# AWQ (pack-quantized)
|
|
240
|
+
xinfer --m cyankiwi/Qwen3-Coder-Next-AWQ-4bit
|
|
241
|
+
|
|
238
242
|
# MultiNode: GLM 5.2 (DeepSeek V3.2 architecture, FP8)
|
|
239
243
|
# Master node
|
|
240
244
|
xinfer --d 0,1,2,3,4,5,6,7 --m zai-org/GLM-5.2-FP8 --num-nodes 2 --node-rank 0 --master-addr 192.168.xxx.xxx
|
package/package.json
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "xinfer-ai",
|
|
3
|
-
"version": "0.
|
|
4
|
-
"assetVersion": "0.
|
|
3
|
+
"version": "0.14.1",
|
|
4
|
+
"assetVersion": "0.14.1",
|
|
5
5
|
"description": "xInfer — a high-performance LLM inference engine in Rust with CUDA/Metal acceleration",
|
|
6
6
|
"license": "MIT",
|
|
7
7
|
"homepage": "https://github.com/guoqingbao/xinfer#readme",
|