xinfer-ai 0.12.1

package/docs/add_model.md

@@ -0,0 +1,88 @@
+# Adding New Model Architectures (AI-Assisted)
+
+xInfer ships with a **Cursor Agent Skill** that guides AI coding assistants through the full workflow of adapting a new HuggingFace model architecture to this project. It covers safetensors and GGUF formats, Dense and MoE architectures, and quantization formats (MXFP4, FP8, ISQ, etc.).
+
+## Prerequisites
+
+- [Cursor IDE](https://cursor.sh/) with Agent mode enabled (for other Agents, mention the skill file manually)
+- The xInfer repository cloned locally
+- (Optional) A local copy of [attention.rs](https://github.com/guoqingbao/attention.rs) if custom kernels are needed
+
+## How It Works
+
+The skill lives at `.cursor/skills/add-model/SKILL.md` and is **automatically activated** when you ask the agent to add, port, or support a new model. It walks the agent through six phases:
+
+| Phase | What happens |
+|-------|-------------|
+| **0 — Gather info** | Collects model config, weight tensor info, and GGUF metadata. Can read from a local model path, HuggingFace Hub, or user-provided files. |
+| **1 — Analyze** | Compares the new architecture against existing models, identifies closest template, classifies as Dense / MoE / Hybrid / Multimodal. |
+| **2 — Kernels** | Determines if new CUDA + Metal kernels are needed in `attention.rs`. If so, clones the repo locally, adds both backends, and points `Cargo.toml` to the local copy. |
+| **3 — Implement** | Creates the model file (`src/models/<arch>.rs`), handles HF and GGUF weight paths, MoE construction, packed expert layouts. |
+| **4 — Register** | Wires the model into `mod.rs`, `config.rs`, `utils/mod.rs`, `runner.rs`, and `parser.rs`. |
+| **5 — Build** | Compiles with the correct platform features and fixes all errors/warnings. |
+| **6 — Test** | Starts the server, sends OpenAI-compatible requests, debugs common issues (NaN, crashes, wrong output). |
+
+## Quick Start
+
+Open the project in Cursor and ask the agent:
+
+```
+Add support for the model google/gemma-4-26B-A4B-it
+```
+
+The agent will read the skill, then ask you for any missing information (config URL, tensor info, etc.). If you have the model downloaded locally, provide the path:
+
+```
+Add support for the model at /path/to/my-model/
+```
+
+The agent will inspect the local weights directly to extract config and tensor info.
+
+## What You Can Provide
+
+The skill accepts multiple input forms — provide whichever is convenient:
+
+- **HuggingFace model ID** — e.g. `google/gemma-4-26B-A4B-it`
+- **Local safetensors directory (prefered)** — path containing `config.json` + `*.safetensors`
+- **Local GGUF file** — a single `.gguf` file
+- **Manual config + tensor info** — paste `config.json` contents (or the path/url) and weight tensor names/shapes (can be obtained by clicking model weights in Hugginface model/files repo)
+
+## Supported Configurations
+
+| Dimension | Supported variants |
+|-----------|--------------------|
+| **Format** | Safetensors (HF), GGUF, GPTQ, AWQ |
+| **Architecture** | Dense, MoE, Hybrid MoE (parallel dense+MoE), Multimodal (vision+language) |
+| **Quantization** | ISQ (in-situ: Q4K, Q8_0, etc.), FP8, MXFP4, NVFP4, GPTQ, AWQ |
+| **Attention** | GQA, MQA, MHA, Sliding Window, Multi-head Latent Attention (MLA) |
+| **Platform** | CUDA (Linux/Windows), Metal (macOS) |
+
+## Multi-GPU Models
+
+For models that require multi-GPU inference:
+
+```bash
+xinfer --m <model_id> --port 8000 --d 0,1
+```
+
+When retrying after a failed load, the agent will kill all `xinfer` processes and verify GPU memory is freed before reloading.
+
+## Debugging Tips
+
+If the model loads but produces incorrect output, the skill guides the agent through:
+
+1. Comparing logits against the Python HuggingFace reference implementation
+2. Checking RoPE parameters (`rope_theta`, `partial_rotary_factor`, `head_dim`)
+3. Verifying activation functions (`Silu` vs `GeluPytorchTanh`)
+4. Inspecting MoE routing (expert selection, normalization, scaling)
+5. Validating weight loading prefixes (HF vs GGUF tensor name mappings)
+
+## File Reference
+
+| File | Role |
+|------|------|
+| `.cursor/skills/add-model/SKILL.md` | The skill definition (read by the AI agent) |
+| `src/models/<arch>.rs` | Model implementation (created by the agent) |
+| `src/models/layers/moe.rs` | Shared MoE layer (FusedMoe, FusedMoeGGUF, ISQ, MXFP4) |
+| `src/utils/mod.rs` | Config parsing and architecture registration (revised by the agent) |
+| `src/core/runner.rs` | Model enum and dispatch macros (revised by the agent) |

package/docs/claude_code.md

@@ -0,0 +1,95 @@
+# Claude Code + xInfer (Anthropic-compatible endpoint)
+
+This guide connects Claude Code directly to xInfer using the built-in
+Anthropic-compatible `/v1/messages` API. No proxy required.
+
+```
+Claude Code -> xInfer (Anthropic-compatible)
+```
+
+## 1) Start xInfer on port 8000
+
+```bash
+# Rust
+xinfer --m Qwen/Qwen3-Coder-Next-FP8 --server --d 0,1
+
+# Different model
+xinfer --m Qwen/Qwen3.5-27B-FP8 --d 0 --server
+
+# Python
+python3 -m xinfer.server --m Qwen/Qwen3-Coder-Next-FP8 --d 0,1
+```
+
+## 2) Configure Claude Code
+
+Install claude code
+
+```shell
+npm install -g @anthropic-ai/claude-code
+```
+
+or
+```shell
+curl -fsSL https://claude.ai/install.sh | bash
+```
+
+Export config
+
+```shell
+export ANTHROPIC_BASE_URL="http://127.0.0.1:8000"
+export ANTHROPIC_AUTH_TOKEN="sk-dummy"
+export CLAUDE_CODE_DISABLE_NONESSENTIAL_TRAFFIC=1
+```
+
+Or make it permanent
+
+Set `~/.claude/settings.json` (or copy from `example/claude/settings.json`):
+
+```json
+{
+  "env": {
+    "ANTHROPIC_BASE_URL": "http://127.0.0.1:8000",
+    "ANTHROPIC_MODEL": "default",
+    "ANTHROPIC_SMALL_FAST_MODEL": "default",
+    "ANTHROPIC_AUTH_TOKEN": "sk-dummy",
+    "CLAUDE_CODE_DISABLE_NONESSENTIAL_TRAFFIC": "1"
+  }
+}
+```
+
+## 3) Run Claude Code
+
+run claude code
+
+```shell
+claude
+```
+
+or verify with a direct request (optional)
+
+```bash
+curl http://127.0.0.1:8000/v1/messages \
+  -H "Content-Type: application/json" \
+  -d '{
+    "model": "default",
+    "max_tokens": 256,
+    "messages": [
+      {"role": "user", "content": "Hello from Claude Code"}
+    ]
+  }'
+```
+
+## Notes
+
+- Streaming uses server-sent events (SSE) on `/v1/messages` with `stream: true`.
+- Token counting is available at `POST /v1/messages/count_tokens`.
+- Embeddings are not part of the Anthropic API and are not exposed here.
+
+### Trouble shooting
+
+1. Use the chat logger to monitor detailed interactions between Claude Code and xInfer.
+
+```shell
+# Log into files (in folder ./log)
+export XINFER_CHAT_LOGGER=1
+```

package/docs/docker.md

@@ -0,0 +1,73 @@
+# xInfer Docker Image
+
+This repository provides a Docker image for **xInfer**, a high-performance inference engine for large language models (LLMs), built using Rust and optimized for NVIDIA GPUs.
+
+---
+
+## Build Options
+
+The image is built with the following features enabled by default:
+
+```text
+cuda,nccl,python,flashinfer,cutlass
+```
+
+CUDA graph capture is auto-enabled with `cuda`; use `--disable-cuda-graph` at runtime to skip it.
+`flashinfer` or `flashattn` increases build time but improves long-context prefill/decoding performance.
+
+For V100, remove `flashattn` or `flashinfer`, and `cutlass`.
+
+For single-GPU machines, you may remove `nccl`.
+
+For SM90+ GPUs, add feature `cutlass` will enable hardware FP8 acceleration.
+
+---
+
+## Docker Build
+
+## Build From Dockerfile
+
+To build this Docker image locally, choose the feature list, compute capability and CUDA version:
+
+Build from script:
+
+```bash
+./build_docker.sh "cuda,nccl,flashinfer,python" sm_80 12.9.0
+```
+
+Build from command line:
+
+```bash
+docker build --network=host -f "Dockerfile" -t "xinfer:latest" \
+  --build-arg CUDA_VERSION="12.9.0" \
+  --build-arg UBUNTU_VERSION="22.04" \
+  --build-arg CUDA_FLAVOR="cudnn-devel" \
+  --build-arg WITH_FEATURES="cuda,nccl,flashinfer,cutlass,python" \
+  --build-arg CUDA_COMPUTE_CAP="sm_80" \
+  # --build-arg CHINA_MIRROR="0" \ Use Rust crate mirror in Chinese mainland
+  .
+```
+
+## Run xInfer docker service
+
+### xInfer Help:
+```bash
+docker run --rm -it --gpus all --network host xinfer:latest xinfer --help
+```
+### Run API server (make sure `--network host`):
+
+```bash
+docker run --rm -it --gpus all --network host xinfer:latest xinfer --m Qwen/Qwen3-0.6B --server
+```
+
+### Run UI + API Server:
+Run interactively:
+```bash
+docker run --rm -it --gpus all --network host -v /home:/home -v /data:/data xinfer:latest bash
+```
+Start the UI + API server
+```bash
+xinfer --w /home/path/Qwen3-Coder-30B-A3B-Instruct-FP8 --ui-server
+```
+
+**Note:** if `Ctrl+C` not working in docker, you need `Ctrl+P` then `Ctrl+Q` to stop the server.

package/docs/embeddings.md

@@ -0,0 +1,33 @@
+# Embedding Usage
+
+This repository now exposes OpenAI-style embeddings for text-only models (Qwen3, Qwen3-MoE, LLaMa, GLM4, Gemma3). Use the standard server run path and hit `/v1/embeddings`.
+
+## Start the server (embeddings enabled)
+- CUDA example (Qwen3 text):  
+  ```bash
+  target/release/xinfer --server --m Qwen/Qwen2.5-7B-Instruct
+  ```
+- Metal example (LLaMa3 text):  
+  ```bash
+  target/release/xinfer --server --m meta-llama/Llama-3-8b --max-model-len 32768
+  ```
+
+## Request examples
+- Float embeddings (default) with mean pooling:  
+  ```bash
+  curl -X POST http://localhost:8000/v1/embeddings \
+    -H "Content-Type: application/json" \
+    -d '{"input":"hello world","model":"default","embedding_type":"mean"}'
+  ```
+- Base64-encoded embeddings with last-token pooling:  
+  ```bash
+  curl -X POST http://localhost:8000/v1/embeddings \
+    -H "Content-Type: application/json" \
+    -d '{"input":["hello","hola"],"embedding_type":"last","encoding_format":"base64"}'
+  ```
+
+## Notes
+- `model` defaults to the loaded model id; multiple models per request are not supported.
+- Uses existing tokenizer; long prompts must fit `max_model_len` (same as chat).
+- `embedding_type`: `mean` (default) averages tokens; `last` returns the final token hidden state.
+- Responses mirror OpenAI schema: `data[].embedding`, `usage.prompt_tokens`.

package/docs/get_started.md

@@ -0,0 +1,93 @@
+# Get Started
+
+This guide walks through building and running xInfer across CUDA/Metal, different model formats, multi-rank, PD Disaggregation, and OpenAI-compatible APIs. Commands assume repo root.
+
+## Build & features
+- **Backends**: `--features cuda[,nccl,graph,flashinfer,cutlass]` or `--features metal`. CPU-only is supported but slow.
+- **Quant/accel toggles**: `--kvcache-dtype fp8|turbo8|turbo4|turbo3` (KV cache quantization), `flashattn` or `flashinfer` (Ampere+), `--prefix-cache` (prefix KV reuse).
+- **Python bindings**: add feature `python` when building wheels (`./build.sh --features python`).
+
+### Build (CUDA)
+```shell
+# Remove `nccl` for single-gpu usage
+# Remove `flashattn`, `flashinfer` and `cutlass` for V100 or older hardware
+./build.sh --release --features cuda,nccl,flashinfer,cutlass
+```
+
+### Build (Metal)
+```shell
+cargo build --release --features metal
+```
+
+## Model formats
+- **Safetensors (HF layout)**: `--m <hf_id>` for cached download, or `--w <local_dir>` for offline weights + configs.
+- **Safetensors (HF layout) + ISQ**: in-situ quantize into GGUF with args `--isq <q4k|q2k|q6k|...>`.
+- **GGUF**: `--f <gguf_file>`; no configs needed.
+- **Vision-Language** (Qwen3-VL, Gemma3, Mistral3-VL): require image tokens; use `--ui-server` for uploads or send image_url/base64 in the request.
+
+
+## 3) Run patterns (single host)
+- **CUDA text model (chat/server)**  
+  ```bash
+  target/release/xinfer --m Qwen/Qwen2.5-7B-Instruct --max-model-len 131072 \
+    --kv-fraction 0.6 --ui-server
+  ```
+- **Metal (Mac) text model**  
+  ```bash
+  target/release/xinfer --m meta-llama/Llama-3-8b --max-model-len 32768 --ui-server
+  ```
+- **GGUF quantized**  
+  ```bash
+  target/release/xinfer --f /path/model-Q4_K_M.gguf --max-model-len 65536  ```
+- **Embeddings** (same server; OpenAI `/v1/embeddings`)  
+  ```bash
+  target/release/xinfer --m Qwen/Qwen2.5-7B-Instruct  # curl -d '{"input":"hello","embedding_type":"mean"}' http://localhost:8000/v1/embeddings
+  ```
+- **Multimodal**  
+  ```bash
+  # Update image in the Chat UI
+  target/release/xinfer --m Qwen/Qwen3-VL-8B-Instruct --ui-server  ```
+
+Common runtime knobs: `--max-model-len`, `--max-num-seqs`, `--kv-fraction` (CUDA KV share), `--cpu-mem-fold` (CPU swap ratio), `--port`, `--kvcache-dtype` (fp8/turbo8/turbo4/turbo3), `--prefix-cache`, `--prefix-cache-max-tokens`, `--ui-server`, `--batch` (perf test).
+
+Reasoning defaults to enabled when a request omits `thinking` / `enable_thinking`. Use `--disable-reasoning` on the Rust CLI to make the default be disabled instead; explicit request values still override the server default.
+
+## 4) Multi-rank (single node)
+- **NCCL multi-GPU**  
+  ```bash
+  target/release/xinfer --m Qwen/Qwen3-30B-A3B-Instruct-2507 --d 0,1 --max-num-seqs 2 --kv-fraction 0.5
+  ```
+- **Graph capture**: CUDA graph is auto-enabled with `cuda` feature. Use `--disable-cuda-graph` at runtime to skip graph capture.
+
+## 5) PD Disaggregation (prefill/decoding split)
+- **PD server (prefill host, usually memory-rich)**  
+  ```bash
+  target/release/xinfer --pd-server --port 8000 \
+    --m Qwen/Qwen3-30B-A3B-Instruct-2507  ```
+- **PD client (decode host)**  
+  ```bash
+  target/release/xinfer --server --pd-client --pd-url 0.0.0.0:8000 \
+    --m Qwen/Qwen3-30B-A3B-Instruct-2507  ```
+- Same weights/config on both ends; Local IPC used automatically on same node CUDA, TCP when `--pd-url` is set. Monitor logs for transfer and swap events.
+
+## Prefix cache
+- Enabled by default (CUDA/Metal). Disable with `--disable-prefix-cache`. Prefix reuse is automatic; no `session_id` required.
+- Use `--prefix-cache-max-tokens` to cap the cache size (rounded down to block size).
+- Tune `--max-model-len`, `--kv-fraction`, `--cpu-mem-fold`; avoid overcommitting KV or cache will swap/evict.
+
+## APIs (OpenAI-style)
+- Chat: `POST /v1/chat/completions` (supports `stream=true`, images for VL models).
+- Embeddings: `POST /v1/embeddings` (`embedding_type=mean|last`, `encoding_format=float|base64`).
+- Models: `GET /v1/models`; Usage: `GET /v1/usage?session_id=...`.
+- UI: add `--ui-server` to expose the built-in web UI on port 8001.
+
+## Troubleshooting & tuning
+- Use `--log` to view loading/progress; watch for “swap” messages (KV pressure).
+- If OOM on Metal, lower `--max-model-len` and batch; on CUDA, reduce `--kv-fraction` or `--max-num-seqs`.
+- For GGUF/ISQ, keep `--max-num-seqs` moderate to avoid bandwidth bottlenecks; `--kvcache-dtype fp8` is supported on all CUDA GPUs (SM70+) and Metal.
+- Use the chat logger to monitor detailed interactions between client and xInfer.
+
+```shell
+# Log into files (in folder ./log)
+export XINFER_CHAT_LOGGER=1
+```

package/docs/goose.md

@@ -0,0 +1,83 @@
+# Goose + xInfer (OpenAI-compatible endpoint)
+
+This guide connects Goose (Rust `AI Agent`) directly to xInfer using the built-in OpenAI-compatible `/v1/chat/completions` API. No proxy required.
+
+```
+Goose -> xInfer (OpenAI-compatible)
+```
+
+## 1) Start xInfer on port 8000
+
+```bash
+# Rust
+xinfer --m Qwen/Qwen3-Coder-Next-FP8 --server --d 0,1
+# Python
+python3 -m xinfer.server --m Qwen/Qwen3-30B-A3B-Instruct-2507 --d 0,1 --server
+```
+
+## 2) Configure Goose
+
+```shell
+# For non-UI system,
+export GOOSE_DISABLE_KEYRING=1
+```
+Export empty API KEY
+
+```shell
+export VLLM_API_KEY="empty"
+```
+
+### Download and install Goose: https://block.github.io/goose/docs/getting-started/installation/
+
+### Configure goose with `Custom Providers` and API key `empty`
+
+```shell
+goose configure
+
+┌   goose-configure
+│
+◇  What would you like to configure?
+│  Custom Providers
+│
+◇  What would you like to do?
+│  Add A Custom Provider
+│
+◇  What type of API is this?
+│  OpenAI Compatible
+│
+◇  What should we call this provider?
+│  xinfer
+│
+◇  Provider API URL:
+│  http://127.0.0.1:8000/v1/
+│
+◇  API key:
+│  ▪▪▪▪▪
+│
+◇  Available models (separate with commas):
+│  default
+│
+◇  Does this provider support streaming responses?
+│  Yes
+│
+◇  Does this provider require custom headers?
+│  No
+│
+└  Custom provider added: xinfer
+└  Configuration saved successfully to /root/.config/goose/config.yaml
+```
+
+### Run `goose` at any folder to work with your local server
+
+```shell
+goose
+```
+
+### Trouble shooting
+
+1. Use the chat logger to monitor detailed interactions between Goose and xInfer.
+
+```shell
+# Log into files (in folder ./log)
+export XINFER_CHAT_LOGGER=1
+```