xinfer-ai 0.12.1

package/docs/multimodal.md

@@ -0,0 +1,37 @@
+# Multimodal Model Usage
+
+This project supports vision-language models (Qwen3-VL dense/MoE, Gemma3, Mistral3-VL). The server exposes `/v1/chat/completions` with mixed text+image content and optional web UI.
+
+## Starting servers
+- Qwen3-VL (CUDA):  
+  ```bash
+  target/release/xinfer --m Qwen/Qwen3-VL-8B-Instruct --ui-server
+  ```
+- Gemma3 (vision, No Flash attention support):  
+  ```bash
+  xinfer --m google/gemma-3-4b-it --ui-server
+  ```
+- Mistral3-VL (vision):  
+  ```bash
+  target/release/xinfer --m mistralai/Ministral-3-8B-Reasoning --ui-server
+  ```
+
+## Request payloads (OpenAI-compatible)
+- Text + image URL:  
+  ```json
+  {
+    "model": "default",
+    "messages": [
+      {"role":"user","content":[
+        {"type":"text","text":"Describe this image"},
+        {"type":"image_url","image_url":"https://example.com/cat.png"}
+      ]}
+    ]
+  }
+  ```
+- Text + base64 image: `{"type":"image_base64","image_base64":"data:image/png;base64,..."}`
+
+## Tips
+- Use smaller `--max-model-len` on Metal if VRAM is tight; consider `--kv-fraction` on CUDA to reserve cache.
+- For batch image inputs, keep concurrent images modest; too many images will increase prefill time.
+- `--ui-server` opens the built-in chat UI for uploads; without it, send HTTP requests directly.***

package/docs/opencode.md

@@ -0,0 +1,83 @@
+# OpenCode + xInfer (OpenAI-compatible endpoint)
+
+This guide connects OpenCode directly to xInfer using the built-in
+OpenAI-compatible `/v1/chat_completions` API. No proxy required.
+
+```
+OpenCode -> xInfer (OpenAI-compatible)
+```
+
+## 1) Start xInfer on port 8000
+
+```bash
+# Rust
+xinfer --m Qwen/Qwen3.5-35B-A3B-FP8 --server --d 0
+
+# Different model
+xinfer --m Qwen/Qwen3.5-27B-FP8 --d 0 --server
+
+# Python
+python3 -m xinfer.server --m Qwen/Qwen3-Coder-Next-FP8 --d 0,1
+```
+
+## 2) Configure OpenCode
+
+Install opencode (CLI)
+
+```shell
+curl -fsSL https://opencode.ai/install | bash
+# Or install with npm
+npm i -g opencode-ai
+```
+
+Export config into `~/.config/opencode/config.json`
+
+
+```json
+{
+  "$schema": "https://opencode.ai/config.json",
+  "provider": {
+    "xinfer": {
+      "npm": "@ai-sdk/openai-compatible",
+      "name": "xInfer Local",
+      "options": {
+        "baseURL": "http://localhost:8000/v1"
+      },
+      "models": {
+        "qwen3-coder": {
+          "name": "Qwen3 Coder"
+        }
+      }
+    }
+  },
+  "model": "xinfer/qwen3-coder"
+}
+```
+
+Install Desktop OpenCode (optional)
+
+```shell
+visit https://opencode.ai/download
+```
+
+Connect to provider -> custom -> base URL (http://localhost:8000/v1) -> Empty key
+
+
+## 3) Run OpenCode
+
+run opencode (CLI)
+
+```shell
+opencode
+```
+
+Or, run OpenCode desktop (choose configured custom provider)
+
+### Trouble shooting
+
+1. Use the chat logger to monitor detailed interactions between OpenCode and xInfer.
+
+```shell
+# Log into files (in folder ./log)
+export XINFER_CHAT_LOGGER=1
+```

package/docs/performance.md

@@ -0,0 +1,65 @@
+# Performance Benchmarks
+
+This document contains detailed performance benchmarks for xInfer across different hardware platforms.
+
+## 🚀 CUDA Performance (A100 40GB)
+
+### Single Request Decoding Speed
+
+| Model | Format | Size | Decoding Speed |
+|-------|--------|------|----------------|
+| Llama-3.1-8B | ISQ (BF16→Q4K) | 8B | **90.19** tokens/s |
+| DeepSeek-R1-Distill-Llama-8B | Q2_K | 8B | **94.47** tokens/s |
+| DeepSeek-R1-0528-Qwen3-8B | Q4_K_M | 8B | **95** tokens/s |
+| GLM-4-9B-0414 | Q4_K_M | 9B | **70.38** tokens/s |
+| QwQ-32B | Q4_K_M | 32B | **35.69** tokens/s |
+| **Qwen3-30B-A3B** | Q4_K_M | **30B (MoE)** | **75.91** tokens/s |
+
+## 🍎 Metal Performance (Apple Silicon M4)
+
+> Test Configuration:
+> - Models: Qwen3-0.6B (BF16), Qwen3-4B (Q4_K_M), Qwen3-8B (Q2_K)
+> - Concurrent Requests: 1 - 128
+> - Max Model Length: 512 - 2048
+> - Max Output Tokens/Request: 512 - 2048
+
+| Model | Batch Size | Output Tokens | Time (s) | Throughput (tokens/s) |
+|-------|------------|---------------|----------|----------------------|
+| Qwen3-0.6B (BF16) | 128 | 63,488 | 83.13s | **763.73** |
+| Qwen3-0.6B (BF16) | 32 | 15,872 | 23.53s | **674.43** |
+| Qwen3-0.6B (BF16) | 1 | 456 | 9.23s | 49.42 |
+| Qwen3-4B (Q4_K_M) | 1 | 1,683 | 52.62s | 31.98 |
+| Qwen3-8B (Q2_K) | 1 | 1,300 | 80.88s | 16.07 |
+
+## 📊 Performance Comparison
+
+> Test Configuration:
+> - Model: Qwen3-0.6B (BF16)
+> - Concurrent Requests: 256
+> - Max Model Length: 1024
+> - Max Output Tokens/Request: 1024
+
+| Inference Engine | Hardware | Tokens | Time (s) | Throughput (tokens/s) |
+|------------------|----------|--------|----------|----------------------|
+| vLLM (Reference) | RTX 4070 | 133,966 | 98.37 | 1,361.84 |
+| Nano-vLLM (Reference) | RTX 4070 | 133,966 | 93.41 | 1,434.13 |
+| **xInfer** | **A100** | 262,144 | 23.88 | **10,977.55** |
+| Nano-vLLM | A100 | 262,144 | 34.22 | 7,660.26 |
+
+### Key Insights
+
+- **40%+ faster** than Nano-vLLM on A100
+- **7x faster** than reference implementations on consumer hardware
+- Efficient memory management with quantized models
+
+## 🔧 Reproduce Benchmarks
+
+See [python/ReadMe.md](../python/ReadMe.md) for reproducible benchmark steps.
+
+## Optimization Tips
+
+1. **Use KV Cache Quantization** (`--kvcache-dtype fp8|turbo4|turbo3`) for memory efficiency — turbo4 gives 3.7x compression with good quality
+2. **Enable Flashinfer / Flash Attention** (`flashinfer` or `flashattn` feature) for maximum CUDA performance
+3. **Prefix Cache** is enabled by default for multi-turn conversations
+4. **Tune `--kv-fraction`** to balance memory usage and batch size
+5. **Use PD Disaggregation** for long-context workloads to prevent decoding stalls

package/docs/prefix-cache.md

@@ -0,0 +1,79 @@
+# Prefix Cache (KV Reuse)
+
+Prefix cache lets xInfer reuse KV cache blocks from prior requests when a new
+prompt shares a prefix. This accelerates consecutive requests with overlapping
+history (for example, chat sessions that replay the same system + earlier turns).
+
+## How it works
+- Finished sequences contribute full KV blocks to a global prefix cache.
+- New requests find the longest cached prefix (block-aligned) and reuse those blocks.
+- Remaining tokens are prefetched as usual, with KV writes continuing after the cached prefix.
+
+Prefix cache is block-granular: only full KV blocks are reused. If the common
+prefix ends mid-block, the tail of that block is recomputed. When a prompt is
+fully cached at block boundaries, the last block is recomputed to ensure a
+non-empty prefill step for correct sampling.
+
+## Flags
+- Prefix cache is **enabled by default**. Use `--disable-prefix-cache` to turn it off.
+- `--prefix-cache-max-tokens <N>`: cap cache size in tokens (rounded down to block size).
+
+If `--prefix-cache-max-tokens` is not set, defaults are:
+- Normal mode: ~50% of GPU KV blocks
+- PD server: ~75% of GPU KV blocks
+- PD client: ~35% of GPU KV blocks
+
+## Hybrid Mamba Snapshot Stride
+For hybrid Mamba models (for example Qwen3.5), prefix reuse also needs a
+compatible Mamba snapshot at the matched boundary.
+
+Use environment variable `XINFER_MAMBA_SNAPSHOT_STRIDE_BLOCKS` to control
+sparse snapshot capture during decode (larger stride side usefull for limited GPU memory):
+- Default: `1` blocks
+- Minimum valid value: `1` (capture every block)
+- Effective snapshot boundary in tokens: `block_size * stride`
+
+Example with default `block_size=64` and stride `8`:
+- Decode snapshot boundary is every `512` tokens.
+- Effective hybrid prefix reuse is aligned to the nearest captured boundary.
+
+This setting only sparsifies decode-time snapshot capture. Prompt/prefill
+snapshot capture remains dense.
+
+## Notes
+- Prefix cache uses the same KV memory pool as active sequences. A larger cache
+  reduces the maximum number of concurrent tokens available for new requests.
+- Cached KV reuse is automatic; no `session_id` is required.
+- Sliding window attention limits how much cached context is effectively used.
+
+## Inspecting cache hits
+
+Chat completion responses include the prefix-cache hit count under
+`usage.prompt_tokens_details.cached_tokens` (OpenAI extension). The field is
+omitted when no hits occurred, so existing single-turn responses keep their
+shape:
+
+```json
+"usage": {
+  "prompt_tokens": 499,
+  "completion_tokens": 16,
+  "total_tokens": 515,
+  "prompt_tokens_details": { "cached_tokens": 480 }
+}
+```
+
+In Python (offline batch), call `engine.get_num_cached_tokens_for_seq(seq_id)`
+on the `seq_id` returned in each `GenerationOutput`.
+
+For models that emit `<think>…</think>` reasoning blocks, responses also
+include `usage.completion_tokens_details.reasoning_tokens` so clients can
+attribute completion cost across reasoning vs final-answer output:
+
+```json
+"usage": {
+  "prompt_tokens": 12,
+  "completion_tokens": 256,
+  "total_tokens": 268,
+  "completion_tokens_details": { "reasoning_tokens": 192 }
+}
+```

package/docs/rust_crate.md

@@ -0,0 +1,119 @@
+# Rust crate usage
+
+This crate exposes a Rust-facing API for loading models, running generation, and optionally running
+an OpenAI-compatible service without changing the existing project structure.
+
+## Add dependency
+
+```toml
+[dependencies]
+xinfer = { git = "https://github.com/guoqingbao/xinfer.git", rev = "1377fa9" }
+
+[features]
+cuda = ["xinfer/cuda"]
+```
+
+Use the same Cargo features you would use for the CLI (`cuda`, `metal`, `nccl`, etc.).
+
+## Direct generation (text)
+
+```rust
+use xinfer::api::{EngineBuilder, ModelRepo};
+use xinfer::server::{ChatMessage, MessageContentType};
+use xinfer::utils::{config::SamplingParams, log_throughput};
+
+fn main() -> anyhow::Result<()> {
+    let mut engine =
+        EngineBuilder::new(ModelRepo::ModelID(("google/gemma-3-4b-it", None))).build()?;
+
+    let messages = vec![ChatMessage {
+        role: "user".to_string(),
+        content: MessageContentType::PureText("Say hello from the Rust API.".to_string()),
+    }];
+
+    let params = SamplingParams::default();
+    let output = engine.generate(params, messages)?;
+    println!("\n\n{}", output.decode_output);
+
+    log_throughput(&vec![output]);
+}
+```
+
+## Multimodal request (URL or base64)
+
+```rust
+use xinfer::api::{EngineBuilder, ModelRepo};
+use xinfer::server::{ChatMessage, MessageContent, MessageContentType};
+use xinfer::utils::config::SamplingParams;
+
+fn main() -> candle_core::Result<()> {
+    let mut engine = EngineBuilder::new(ModelRepo::ModelID((
+        "Qwen/Qwen3-VL-8B-Instruct".to_string(),
+        None,
+    )))
+    .build()?;
+
+    let messages = vec![ChatMessage {
+        role: "user".to_string(),
+        content: MessageContentType::Multi(vec![
+            MessageContent::Text {
+                text: "Describe this image:".to_string(),
+            },
+            MessageContent::ImageUrl {
+                image_url: "https://example.com/cat.png".to_string(),
+            },
+        ]),
+    }];
+
+    let params = SamplingParams::default();
+    let output = engine.generate(params, messages)?;
+    println!("{}", output.decode_output);
+
+    Ok(())
+}
+```
+
+## Serve API
+
+```rust
+use xinfer::api::{EngineBuilder, ModelRepo};
+
+fn main() -> candle_core::Result<()> {
+    let mut engine = EngineBuilder::new(ModelRepo::ModelID((
+        "Qwen/Qwen3-0.6B".to_string(),
+        None,
+    )))
+    .build()?;
+
+    engine.start_server(8000, true, false)?;
+    Ok(())
+}
+```
+
+## Multi-rank / multi-GPU
+
+Provide `device_ids` with `with_multirank` (e.g. `"0,1"`) along with the same CUDA/NCCL features
+you use for the CLI. The Rust API reuses the same engine and scheduler path.
+
+```rust
+use xinfer::api::{EngineBuilder, ModelRepo};
+
+fn main() -> candle_core::Result<()> {
+    let mut engine = EngineBuilder::new(ModelRepo::ModelFile(vec![
+        "/path/Qwen3-VL-8B-Instruct-GGUF-Q4_KM.gguf".to_string(),
+    ]))
+    .with_multirank("0,1")?
+    .build()?;
+
+    engine.start_server(8000, true, true)?;
+    Ok(())
+}
+```
+## Command to run
+
+[Reference Rust demo](/example/rust-demo/)
+
+```shell
+# add `nccl` feature for multirank inference
+cargo run --release --features cuda -- ARGS
+```

package/docs/test_model.md

@@ -0,0 +1,104 @@
+# Model Testing (AI-Assisted)
+
+xInfer ships with a **Cursor Agent Skill** that automates the process of testing LLM models for correctness, output quality, and inference performance. It handles GPU detection, model loading, API testing, and result summarization.
+
+## Prerequisites
+
+- [Cursor IDE](https://cursor.sh/) with Agent mode enabled (for other Agents, mention the skill file manually)
+- The xInfer repository cloned locally
+- One or more NVIDIA GPUs with sufficient memory for the target models
+- Python 3 with the requests library installed
+
+## How It Works
+
+The skill lives at .cursor/skills/test-model/SKILL.md and is **automatically activated** when you ask the agent to test, benchmark, or validate models. It walks the agent through five phases:
+
+| Phase | What happens |
+|-------|-------------|
+| **0 - Gather models** | Collects model list from a local folder scan or user-provided HuggingFace IDs. Auto-detects supported architectures and quantization formats. |
+| **1 - Estimate resources** | Queries nvidia-smi for available GPUs and free memory. Estimates per-model memory requirements and assigns GPUs accordingly. |
+| **2 - Build** | Compiles the project with `./build.sh --install --features cuda,nccl,flashinfer,cutlass`. |
+| **3 - Create test script** | Writes test_model.py, an OpenAI-compatible API test that sends 1k+ input tokens, requests 1k+ output tokens, and measures throughput and quality. |
+| **4 - Test each model** | Iteratively starts the server, waits for readiness, runs the test script (with and without reasoning/thinking), records results, kills the server, and moves to the next model. |
+| **5 - Summarize** | Produces a markdown table with per-model results: format, GPUs, throughput, and quality verdict. |
+
+## Quick Start
+
+Open the project in Cursor and ask the agent:
+
+```
+Test all models in /data/
+```
+
+The agent will scan the directory, identify compatible models, and test each one automatically.
+
+You can also specify individual models:
+
+```
+Test models AxionML/Qwen3.5-2B-NVFP4, Qwen/Qwen3-4B
+```
+
+Or mix local paths and HuggingFace IDs:
+
+```
+Test /data/Qwen3.5-27B-FP8 and AxionML/Qwen3.5-9B-NVFP4
+```
+
+## What Gets Tested
+
+For each model, the skill tests:
+
+| Aspect | Details |
+|--------|---------|
+| **Loading** | Server starts and model is accessible via /v1/models |
+| **Inference (no thinking)** | 1k+ input tokens, 2k output tokens, thinking=false |
+| **Inference (with thinking)** | Same prompt with thinking=true / reasoning enabled |
+| **Output quality** | Coherence check, repetition detection (3-gram analysis) |
+| **Performance** | End-to-end throughput in tokens/second |
+
+## Supported Model Formats
+
+| Format | Detection method |
+|--------|-----------------|
+| **BF16 / FP16** (safetensors) | No quantization_config in config.json |
+| **FP8** (blockwise) | quant_method: fp8 |
+| **MXFP4** | quant_method: mxfp4 or format contains mxfp4 |
+| **NVFP4** (modelopt) | quant_method: modelopt with quant_algo: NVFP4 |
+| **NVFP4** (compressed-tensors) | format contains nvfp4 |
+| **GGUF** | .gguf file present in directory |
+| **GPTQ / AWQ** | quant_method: gptq or awq |
+
+## GPU Assignment
+
+The agent automatically assigns GPUs based on available memory:
+
+- **Single GPU**: Models that fit in one GPU's free memory
+- **Multi-GPU**: Large models that require 2+ GPUs (uses --d 0,1 etc.)
+- **Skip**: Models that exceed all available GPU memory
+
+## Debugging Failures
+
+If a model fails to load or produce output, the skill guides the agent through:
+
+1. Checking server logs for panics or weight-loading errors
+2. Enabling RUST_BACKTRACE=1 for stack traces
+3. Temporarily using guard.step().unwrap() in engine.rs for crash debugging
+4. Identifying common issues (MLX format incompatibility, quantization detection, OOM)
+
+## Example Output (Decoding performance)
+
+| # | Model | Format | GPUs | thinking=false | thinking=true | Quality |
+|---|-------|--------|------|----------------|---------------|---------|
+| 1 | Qwen3-4B-FP8 | FP8 | 1 | 1329 in / 2048 out, 168 tok/s | 1329 in / 2048 out, 169 tok/s | OK |
+| 2 | Qwen3.5-27B | BF16 | 1 | 1342 in / 2048 out, 28 tok/s | 1342 in / 2048 out, 27 tok/s | OK |
+| 3 | Qwen3.5-35B-A3B-GGUF | Q3_K_M | 1 | 1342 in / 2048 out, 95 tok/s | 1342 in / 2048 out, 98 tok/s | OK |
+
+## File Reference
+
+| File | Role |
+|------|------|
+| .cursor/skills/test-model/SKILL.md | The skill definition (read by the AI agent) |
+| test_model.py | OpenAI API test script (created by the skill) |
+| build.sh | Build script for the xinfer binary |
+| src/core/engine.rs | Engine loop, guard.step() location for debug |
+| src/models/layers/deltanet.rs | DeltaNet layer with per-weight quantization detection |

package/docs/tokenizer_api.md

@@ -0,0 +1,116 @@
+# Tokenizer API
+
+The Tokenizer API provides direct access to the model's tokenizer for encoding/decoding text without running inference. This is useful for:
+
+- Pre-computing token counts for cost estimation
+- Validating inputs before sending to the model
+- Debugging tokenization issues
+- Building custom tooling around the tokenizer
+
+## Endpoints
+
+### POST /tokenize
+
+Convert text or chat messages to token IDs.
+
+**Request (plain text):**
+```json
+{
+  "prompt": "Hello, world!"
+}
+```
+
+**Request (chat messages - applies chat template):**
+```json
+{
+  "messages": [
+    {"role": "system", "content": "You are helpful."},
+    {"role": "user", "content": "Hello!"}
+  ]
+}
+```
+
+**Optional parameters:**
+- `model`: Model name (optional, uses loaded model)
+- `add_special_tokens`: Whether to add special tokens (default: `true`)
+
+**Response:**
+```json
+{
+  "tokens": [1, 2, 3, 4],
+  "count": 4,
+  "max_model_len": 4096
+}
+```
+
+### POST /detokenize
+
+Convert token IDs back to text.
+
+**Request:**
+```json
+{
+  "tokens": [1, 2, 3, 4]
+}
+```
+
+**Optional parameters:**
+- `model`: Model name (optional, uses loaded model)
+- `skip_special_tokens`: Whether to skip special tokens in output (default: `true`)
+
+**Response:**
+```json
+{
+  "prompt": "Hello, world!"
+}
+```
+
+## Examples
+
+### Python
+
+```python
+import requests
+
+# Tokenize text
+response = requests.post("http://localhost:8000/tokenize", json={
+    "prompt": "Hello, world!"
+})
+print(response.json())
+# {"tokens": [9906, 11, 1917, 0], "count": 4, "max_model_len": 4096}
+
+# Detokenize
+response = requests.post("http://localhost:8000/detokenize", json={
+    "tokens": [9906, 11, 1917, 0]
+})
+print(response.json())
+# {"prompt": "Hello, world!"}
+```
+
+### cURL
+
+```bash
+# Tokenize
+curl -X POST http://localhost:8000/tokenize \
+  -H "Content-Type: application/json" \
+  -d '{"prompt": "Hello, world!"}'
+
+# Detokenize
+curl -X POST http://localhost:8000/detokenize \
+  -H "Content-Type: application/json" \
+  -d '{"tokens": [9906, 11, 1917, 0]}'
+```
+
+## Demo Scripts
+
+- **Python**: `example/tokenize.py` - Interactive demo with multiple examples
+- **Rust**: `example/rust-demo-tokenize/` - Rust client demo
+
+Run the Python demo:
+```bash
+# Start the server first
+cargo run --release -- --m Qwen/Qwen2.5-0.5B-Instruct --server
+
+# In another terminal
+python example/tokenize.py
+```

package/docs/tool_parsing.md

@@ -0,0 +1,74 @@
+## Tool Call Parsing
+
+This project uses the model-specific parsers for parsing tool calls for both
+streaming and non-streaming responses. The goal is to keep parsing logic
+consistent across models while remaining robust to partial output and format
+differences.
+
+### Parser selection
+
+Parser selection follows this order:
+
+1. `--enforce-parser` (if provided and valid).
+2. Model-based heuristics (model type + model id).
+3. Fallback to `passthrough`.
+
+If you pass an invalid name to `--enforce-parser`, the server returns an error
+and includes the list of valid parser names.
+
+Available parser names:
+
+- passthrough
+- json
+- mistral
+- qwen
+- qwen_coder
+- pythonic
+- llama
+- deepseek
+- glm45_moe
+- glm47_moe
+- step3
+- kimik2
+- minimax_m2
+
+### Streaming parsing
+
+Streaming requests use incremental parsing logic. Each incoming token is appended to an internal buffer and fed into the parser. The parser emits streaming tool call fragments, which are accumulated into full tool calls.
+
+When an end marker is detected (token id or `</tool_call>` tag), the stream parser:
+
+1. Flushes any unstreamed arguments from the external parser.
+2. Builds tool calls from the accumulated fragments.
+3. If no tool calls were produced, falls back to `parse_complete_with_fallback`
+   on the buffered content.
+
+If parsing still fails, the buffered content is emitted as normal text so the
+client does not lose output.
+
+### Non-streaming parsing
+
+Non-streaming requests reuse the same stream parser and call
+`parse_complete_with_fallback`. This keeps parser selection and fallback logic
+identical between streaming and non-streaming paths.
+
+### Enforcing a parser
+
+CLI (Rust server):
+
+```
+--enforce-parser qwen_coder
+```
+
+Python server example (`server.py` or `xinfer.server`):
+
+```
+--enforce-parser qwen_coder
+```
+
+### Environment Variables
+
+- `XINFER_STRICT_TOOL_CALL`:
+  - `1` or `true`: Strict validation. Dropping invalid tool calls (calls that do not match the schema) effectively preventing them from being sent to the client. The server logs a warning for dropped calls.
+  - `0` or `false` (default): Lenient validation. Invalid tool calls are kept and sent to the client, but a warning is logged by the server. This allows models to output "hallucinated" or malformed calls if desired.
+