xinfer-ai 0.12.4 → 0.12.6

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (3) hide show
  1. package/README.md +2 -0
  2. package/docs/performance.md +44 -40
  3. package/package.json +2 -2
package/README.md CHANGED
@@ -16,6 +16,7 @@
16
16
  | **🌍** | Cross-platform | CUDA (Linux/Windows), Metal (macOS). Same binary, same API |
17
17
  | **🏭** | Production-ready | OpenAI/Anthropic-compatible APIs, built-in ChatGPT-style Web UI, MCP tool calling, structured outputs, embedding + tokenizer endpoints |
18
18
  | **🗜️** | Aggressive KV compression | TurboQuant (`2–4 bit` KV cache) extends context up to **4.3×** with minimal quality loss. Run `30B+` MoE models with **millions of context** on single 24/32 GB GPUs |
19
+ | **🔥** | V100 + NVFP4 | First-ever NVFP4 + low-bit KV cache on V100 — no hardware FP4 needed, coherent output on legacy GPUs |
19
20
  | **🐍** | Lightweight Python bindings | Optional PyO3 wheel when you need a Python entry point |
20
21
 
21
22
  ---
@@ -446,6 +447,7 @@ Constraint-based generation via llguidance — Lark grammars, regex, JSON Schema
446
447
  | `--pd-url` | PD connection URL (`tcp://`, `http://`, `file://`) |
447
448
  | `--disable-prefix-cache` | Disable prefix caching |
448
449
  | `--prefix-cache-max-tokens` | Cap prefix cache size |
450
+ | `--prefill-chunk-size` | Cap prefill chunk size (default: CUDA 8K, Metal: 4k) |
449
451
  | `--disable-cuda-graph` | Disable CUDA graph capture |
450
452
  | `--yarn-scaling-factor` | YARN RoPE context extension factor |
451
453
  | `--temperature` | Sampling temperature (0–1) |
package/docs/performance.md CHANGED
@@ -2,27 +2,41 @@
2
2
 
3
3
  This document contains detailed performance benchmarks for xInfer across different hardware platforms.
4
4
 
5
- ## 🚀 CUDA Performance (A100 40GB)
5
+ ## 🚀 CUDA Performance
6
+
7
+ > Tested on **V100-32G**, **A100-40G**, **Hopper-80G** and **RTX 5090**
6
8
 
7
9
  ### Single Request Decoding Speed
8
10
 
9
- | Model | Format | Size | Decoding Speed |
10
- |-------|--------|------|----------------|
11
- | Llama-3.1-8B | ISQ (BF16→Q4K) | 8B | **90.19** tokens/s |
12
- | DeepSeek-R1-Distill-Llama-8B | Q2_K | 8B | **94.47** tokens/s |
13
- | DeepSeek-R1-0528-Qwen3-8B | Q4_K_M | 8B | **95** tokens/s |
14
- | GLM-4-9B-0414 | Q4_K_M | 9B | **70.38** tokens/s |
15
- | QwQ-32B | Q4_K_M | 32B | **35.69** tokens/s |
16
- | **Qwen3-30B-A3B** | Q4_K_M | **30B (MoE)** | **75.91** tokens/s |
11
+ | Model | Format | Size | Hardware | Decoding Speed |
12
+ |-------|--------|------|----------|----------------|
13
+ | Qwen3-30B-A3B | NVFP4 | 30B MoE | RTX 5090 (SM120) | **175.30** tokens/s |
14
+ | Gemma4-26B-A4B | NVFP4 | 26B MoE | RTX 5090 (SM120) | **131** tokens/s |
15
+ | Ministral-3-3B (Multimodal) | ISQ (BF16→Q4K) | 3B | A100 (SM80) | **171.92** tokens/s |
16
+ | DeepSeek-R1-0528-Qwen3-8B | Q4_K_M | 8B | A100 (SM80) | **124.87** tokens/s |
17
+ | Llama-3.1-8B | ISQ (BF16→Q4K) | 8B | A100 (SM80) | **120.74** tokens/s |
18
+ | Qwen3-VL-8B-Instruct (Multimodal) | Q8_0 | 8B | A100 (SM80) | **105.31** tokens/s |
19
+ | Qwen3.6-35B-A3B (Multimodal) | FP8 | 35B MoE | H800 (SM90) | **102** tokens/s |
20
+ | GLM4.7 Flash | NVFP4 | 30B MoE | H800 (SM90) | **79** tokens/s |
21
+ | Qwen3-30B-A3B | NVFP4 | 30B MoE | V100 (SM70) | **67.10** tokens/s |
22
+ | GLM-4-9B-0414 | Q4_K_M | 9B | A100 (SM80) | **70.38** tokens/s |
23
+ | MiniMax-M2.5 | NVFP4 | 229B MoE | H800 ×2 (SM90) | **62** tokens/s |
24
+ | Qwen3.5-27B (Multimodal) | Q4_K_M | 27B Dense | A100 (SM80) | **45.20** tokens/s |
25
+ | Qwen3.5-27B/Qwen3.6-27B | FP8 | 27B Dense | H800 (SM90) | **42** tokens/s |
26
+ | QwQ-32B | Q4_K_M | 32B | A100 (SM80) | **41.36** tokens/s |
27
+ | Gemma4-31B | ISQ (BF16→Q4K) | 31B Dense | H800 (SM90) | **41** tokens/s |
28
+
29
+ ### V100 + NVFP4 + TurboQuant (First-Ever)
30
+
31
+ NVFP4 models running under low-bit KV cache on V100 (SM70) — no hardware FP4 support needed.
32
+
33
+ ```bash
34
+ xinfer --m AxionML/Qwen3.5-4B-NVFP4 --ui-server
35
+ xinfer --m AxionML/Qwen3.5-4B-NVFP4 --kvcache-dtype turbo4 --ui-server
36
+ ```
17
37
 
18
38
  ## 🍎 Metal Performance (Apple Silicon M4)
19
39
 
20
- > Test Configuration:
21
- > - Models: Qwen3-0.6B (BF16), Qwen3-4B (Q4_K_M), Qwen3-8B (Q2_K)
22
- > - Concurrent Requests: 1 - 128
23
- > - Max Model Length: 512 - 2048
24
- > - Max Output Tokens/Request: 512 - 2048
25
-
26
40
  | Model | Batch Size | Output Tokens | Time (s) | Throughput (tokens/s) |
27
41
  |-------|------------|---------------|----------|----------------------|
28
42
  | Qwen3-0.6B (BF16) | 128 | 63,488 | 83.13s | **763.73** |
@@ -30,36 +44,26 @@ This document contains detailed performance benchmarks for xInfer across differe
30
44
  | Qwen3-0.6B (BF16) | 1 | 456 | 9.23s | 49.42 |
31
45
  | Qwen3-4B (Q4_K_M) | 1 | 1,683 | 52.62s | 31.98 |
32
46
  | Qwen3-8B (Q2_K) | 1 | 1,300 | 80.88s | 16.07 |
47
+ | Qwen3.5-4B (Q3_K_M) | 1 | 1,592 | 69.04s | 23.06 |
48
+ | Qwen3.5-2B (NVFP4) | 1 | 1,883 | 60.76s | 30.99 |
49
+ | Qwen3.5-2B (NVFP4) | 2 | 3,942 | 81.96s | 48.10 |
33
50
 
34
- ## 📊 Performance Comparison
35
-
36
- > Test Configuration:
37
- > - Model: Qwen3-0.6B (BF16)
38
- > - Concurrent Requests: 256
39
- > - Max Model Length: 1024
40
- > - Max Output Tokens/Request: 1024
51
+ ## 📊 Performance Notes
41
52
 
42
- | Inference Engine | Hardware | Tokens | Time (s) | Throughput (tokens/s) |
43
- |------------------|----------|--------|----------|----------------------|
44
- | vLLM (Reference) | RTX 4070 | 133,966 | 98.37 | 1,361.84 |
45
- | Nano-vLLM (Reference) | RTX 4070 | 133,966 | 93.41 | 1,434.13 |
46
- | **xInfer** | **A100** | 262,144 | 23.88 | **10,977.55** |
47
- | Nano-vLLM | A100 | 262,144 | 34.22 | 7,660.26 |
53
+ - **HW NVFP4**: Hardware-accelerated FP4 on Blackwell (SM120, RTX 5090/B200)
54
+ - **HW FP8**: Hardware-accelerated FP8 on Hopper (SM90, H800/H200)
55
+ - **SW FP4/NVFP4**: Software-emulated FP4 on Hopper (SM90) and V100 (SM70)
56
+ - V100 + NVFP4 + TurboQuant is a first-ever combination — no other engine has achieved this
48
57
 
49
- ### Key Insights
58
+ ## 🔧 Optimization Tips
50
59
 
51
- - **40%+ faster** than Nano-vLLM on A100
52
- - **7x faster** than reference implementations on consumer hardware
53
- - Efficient memory management with quantized models
60
+ 1. **Use KV Cache Quantization** (`--kvcache-dtype fp8|turbo8|turbo4|turbo3`) for memory efficiency — turbo4 gives 3.7× compression with good quality
61
+ 2. **Enable FlashInfer** (`flashinfer` feature, SM80+) for maximum CUDA performance
62
+ 3. **Prefix Cache** is enabled by default for multi-turn conversations
63
+ 4. **Tune `--kv-fraction`** to balance memory usage and batch size
64
+ 5. **Use PD Disaggregation** for long-context workloads to prevent decoding stalls
65
+ 6. **NVFP4 on V100** — works with TurboQuant KV cache for extended context on legacy hardware
54
66
 
55
67
  ## 🔧 Reproduce Benchmarks
56
68
 
57
69
  See [python/ReadMe.md](../python/ReadMe.md) for reproducible benchmark steps.
58
-
59
- ## Optimization Tips
60
-
61
- 1. **Use KV Cache Quantization** (`--kvcache-dtype fp8|turbo4|turbo3`) for memory efficiency — turbo4 gives 3.7x compression with good quality
62
- 2. **Enable Flashinfer / Flash Attention** (`flashinfer` or `flashattn` feature) for maximum CUDA performance
63
- 3. **Prefix Cache** is enabled by default for multi-turn conversations
64
- 4. **Tune `--kv-fraction`** to balance memory usage and batch size
65
- 5. **Use PD Disaggregation** for long-context workloads to prevent decoding stalls
package/package.json CHANGED
@@ -1,7 +1,7 @@
1
1
  {
2
2
  "name": "xinfer-ai",
3
- "version": "0.12.4",
4
- "assetVersion": "0.12.4",
3
+ "version": "0.12.6",
4
+ "assetVersion": "0.12.6",
5
5
  "description": "xInfer — a high-performance LLM inference engine in Rust with CUDA/Metal acceleration",
6
6
  "license": "MIT",
7
7
  "homepage": "https://github.com/guoqingbao/xinfer#readme",