xinfer-ai 0.13.3 → 0.13.4

package/README.md

@@ -338,6 +338,24 @@ See [more Python examples →](python/ReadMe.md)
 
 ---
 
+## 🌐 Multi-Node Tensor Parallelism
+
+Distribute inference across multiple machines using TCP-based NCCL bootstrap. No MPI required.
+
+```bash
+# Node 0 (master, 192.168.1.100): scheduler + API
+xinfer --d 0,1,2,3 --m /data/DeepSeek-R1/ \
+  --num-nodes 2 --node-rank 0 --master-addr 192.168.1.100 --ui-server
+
+# Node 1 (worker, 192.168.1.101): forward-only daemon
+xinfer --d 0,1,2,3 --m /data/DeepSeek-R1/ \
+  --num-nodes 2 --node-rank 1 --master-addr 192.168.1.100
+```
+
+All nodes must have model weights locally and be TCP-reachable on `--master-port` (default 29500). See [Get Started](docs/get_started.md) for details.
+
+---
+
 ## 🔀 Prefill-Decode Disaggregation
 
 Split prefill (prompt processing) and decode (token generation) across GPUs or machines. Eliminates decode stalls during long-context prefilling. PD Server and PD Client must use **same** KvCache type (`--kvcache-dtype`). API request(s) must send to PD Client and the PD Server only process internal prefill requests sent from PD Client.
@@ -437,7 +455,7 @@ Constraint-based generation via llguidance — Lark grammars, regex, JSON Schema
 | `--f` | Local GGUF file when used alone; remote GGUF filename when paired with `--m <model_id>` |
 | `--d` | Device IDs (e.g. `--d 0,1`) |
 | `--ui-server` | API server + built-in ChatGPT-style web UI |
-| `--server` | API server only (no web UI) |
+| `--server` | API server. Bare: `0.0.0.0:8000`; or `host[:port]`, `file:///path`, `socket:///path`, `unix:///path` |
 | `--i` | Interactive CLI chat |
 | `--isq` | On-the-fly quantization: `q2k`, `q3k`, `q4k`, `q5k`, `q6k`, `q8_0` |
 | `--kvcache-dtype` | KV cache quantization: `fp8`, `turbo8`, `turbo4`, `turbo3` |
@@ -488,6 +506,7 @@ XINFER_NVFP4_FORCE_LUT=1 xinfer --m nvidia/Qwen3-30B-A3B-FP4 --ui-server
 * [x] OpenAI-compatible API (streaming support)
 * [x] Continuous batching
 * [x] Multi-gpu inference (Safetensors, GPTQ, AWQ, GGUF)
+* [x] Multi-node tensor parallelism (TCP-based NCCL bootstrap, no MPI required)
 * [x] Speedup prompt processing on Metal/macOS
 * [x] Chunked Prefill
 * [x] Prefix cache (available on `CUDA` when `prefix-cache` enabled)

package/docs/get_started.md

@@ -65,6 +65,52 @@ Reasoning defaults to enabled when a request omits `thinking` / `enable_thinking
   ```
 - **Graph capture**: CUDA graph is auto-enabled with `cuda` feature. Use `--disable-cuda-graph` at runtime to skip graph capture.
 
+## 4.1) Multi-node tensor parallelism
+
+Distribute tensor-parallel inference across multiple machines via TCP-based NCCL bootstrap. Node 0 is the coordinator (runs scheduler + API); worker nodes run forward-only daemon loops.
+
+**Requirements:**
+- All nodes must have the same model weights available locally (same path or HuggingFace cache).
+- All nodes must be reachable via TCP on the `--master-port` (default 29500) and `--master-port + 1` (forward coordination).
+- Build with `--features cuda,nccl,flashinfer,cutlass` on all nodes.
+
+**Example: 2 nodes × 4 GPUs = 8-way tensor parallelism**
+
+```bash
+# Node 0 (master, at 192.168.1.100): runs scheduler + API server
+xinfer --m /data/DeepSeek-R1/ --d 0,1,2,3 \
+  --num-nodes 2 --node-rank 0 \
+  --master-addr 192.168.1.100 --master-port 29500 \
+  --ui-server
+
+# Node 1 (worker, at 192.168.1.101): runs forward-only daemon
+xinfer --m /data/DeepSeek-R1/ --d 0,1,2,3 \
+  --num-nodes 2 --node-rank 1 \
+  --master-addr 192.168.1.100 --master-port 29500
+```
+
+**How it works:**
+1. Node 0 generates a NCCL unique ID and distributes it to worker nodes via TCP.
+2. Each node spawns local runner subprocesses with global NCCL ranks (`node_rank × local_gpus + local_rank`).
+3. All ranks join a single global NCCL communicator for all-reduce / all-gather operations.
+4. Node 0 broadcasts forward-pass commands to worker nodes via TCP; NCCL synchronizes the actual tensor computations.
+5. Only node 0 runs the scheduler and API server; worker nodes are stateless forward engines.
+
+**CLI flags:**
+| Flag | Default | Description |
+|------|---------|-------------|
+| `--num-nodes` | 1 | Total number of nodes |
+| `--node-rank` | 0 | This node's rank (0 = master) |
+| `--master-addr` | _(required)_ | Master node's IP address |
+| `--master-port` | 29500 | TCP port for NCCL bootstrap |
+
+**NCCL environment variables** (optional tuning):
+```bash
+export NCCL_IB_DISABLE=1      # Disable InfiniBand if unavailable
+export NCCL_SOCKET_IFNAME=eth0 # Specify network interface
+export NCCL_DEBUG=INFO          # Enable NCCL debug logging
+```
+
 ## 5) PD Disaggregation (prefill/decoding split)
 - **PD server (prefill host, usually memory-rich)**  
   ```bash

package/package.json

@@ -1,7 +1,7 @@
 {
   "name": "xinfer-ai",
-  "version": "0.13.3",
-  "assetVersion": "0.13.3",
+  "version": "0.13.4",
+  "assetVersion": "0.13.4",
   "description": "xInfer — a high-performance LLM inference engine in Rust with CUDA/Metal acceleration",
   "license": "MIT",
   "homepage": "https://github.com/guoqingbao/xinfer#readme",