xinfer-ai 0.12.9 → 0.13.1

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (3) hide show
  1. package/README.md +2 -2
  2. package/docs/performance.md +1 -1
  3. package/package.json +2 -2
package/README.md CHANGED
@@ -11,7 +11,7 @@
11
11
  | | Feature | Details |
12
12
  |---|---|---|
13
13
  | **0️⃣** | Zero Python dependencies | Pure Rust backend — no PyTorch, no CUDA Python bindings |
14
- | **⚡** | Fast | Native Flash Attention, FlashInfer, CUDA Graphs, continuous batching, prefix caching, PD disaggregation. Up to **181 tok/s** decode for `30B+` models on consumer GPUs |
14
+ | **⚡** | Fast | Native Flash Attention, FlashInfer, CUDA Graphs, continuous batching, prefix caching, PD disaggregation. Up to **197 tok/s** decode for `30B+` models on consumer GPUs |
15
15
  | **🪶** | Tiny footprint | Core scheduling + attention logic in **< 5 000 lines** of Rust |
16
16
  | **🌍** | Cross-platform | CUDA (Linux/Windows), Metal (macOS). Same binary, same API |
17
17
  | **🏭** | Production-ready | OpenAI/Anthropic-compatible APIs, built-in ChatGPT-style Web UI, MCP tool calling, structured outputs, embedding + tokenizer endpoints |
@@ -85,7 +85,7 @@ Add `--kvcache-dtype` to compress KV cache and extend context length:
85
85
  | DeepSeek-R1-0528-Qwen3-8B | Q4_K_M | 8B | **139.25** tokens/s |
86
86
  | GLM-4-9B-0414 | Q4_K_M | 9B | **77.48** tokens/s |
87
87
  | QwQ-32B | Q4_K_M | 32B | **46.02** tokens/s |
88
- | **Qwen3-30B-A3B** | NVFP4 | **30B (MoE)** | **181.59** tokens/s (**RTX 5090**) |
88
+ | **Qwen3-30B-A3B** | NVFP4 | **30B (MoE)** | **197.29** tokens/s (**RTX 5090**) |
89
89
  | **Qwen3-30B-A3B** | NVFP4 | **30B (MoE)** | **72.86** tokens/s (**V100, Software FP4**) |
90
90
  | **Qwen3.5-27B** (**Multimodal**) | Q4_K_M | **27B (Dense)** | **49.33** tokens/s |
91
91
  | **Qwen3.5-27B/Qwen3.6-27B** | FP8 | **27B (Dense)** | **45** tokens/s (**Hopper**) |
package/docs/performance.md CHANGED
@@ -16,7 +16,7 @@ This document contains detailed performance benchmarks for xInfer across differe
16
16
  | DeepSeek-R1-0528-Qwen3-8B | Q4_K_M | 8B | A100| **139.25** tokens/s |
17
17
  | GLM-4-9B-0414 | Q4_K_M | 9B | A100| **77.48** tokens/s |
18
18
  | QwQ-32B | Q4_K_M | 32B | A100| **46.02** tokens/s |
19
- | **Qwen3-30B-A3B** | NVFP4 | **30B (MoE)** | RTX 5090 | **181.59** tokens/s|
19
+ | **Qwen3-30B-A3B** | NVFP4 | **30B (MoE)** | RTX 5090 | **197.29** tokens/s|
20
20
  | **Qwen3-30B-A3B** | NVFP4 | **30B (MoE)** | V100 | **72.86** tokens/s (**Software FP4**) |
21
21
  | **Qwen3.5-27B** (**Multimodal**) | Q4_K_M | **27B (Dense)** | Hopper | **49.33** tokens/s |
22
22
  | **Qwen3.5-27B/Qwen3.6-27B** | FP8 | **27B (Dense)** | Hopper | **45** tokens/s|
package/package.json CHANGED
@@ -1,7 +1,7 @@
1
1
  {
2
2
  "name": "xinfer-ai",
3
- "version": "0.12.9",
4
- "assetVersion": "0.12.9",
3
+ "version": "0.13.1",
4
+ "assetVersion": "0.13.1",
5
5
  "description": "xInfer — a high-performance LLM inference engine in Rust with CUDA/Metal acceleration",
6
6
  "license": "MIT",
7
7
  "homepage": "https://github.com/guoqingbao/xinfer#readme",