pi-usereq 0.4.0

package/docs/pi.dev/pods-docs/models.md

@@ -0,0 +1,116 @@
+### Qwen-Coder
+- [ ] Qwen2.5-Coder-32B-Instruct
+  - HF: Qwen/Qwen2.5-Coder-32B-Instruct
+  - Hardware:
+    - 1x H100/H200
+      - --tool-call-parser hermes --enable-auto-tool-choice
+    - 2x H100/H200
+      - --tensor-parallel-size 2 --tool-call-parser hermes --enable-auto-tool-choice
+  - Notes: Good balance of size and performance. Single GPU capable.
+- [ ] Qwen3-Coder-480B-A35B-Instruct (BF16)
+  - HF: Qwen/Qwen3-Coder-480B-A35B-Instruct
+  - Hardware:
+    - 8x H200/H20
+      - --tensor-parallel-size 8 --max-model-len 32000 --enable-auto-tool-choice --tool-call-parser qwen3_coder
+      - Notes: Cannot serve full 262K context on single node. Reduce max-model-len or increase gpu-memory-utilization.
+- [ ] Qwen3-Coder-480B-A35B-Instruct-FP8
+  - HF: Qwen/Qwen3-Coder-480B-A35B-Instruct-FP8
+  - Hardware:
+    - 8x H200/H20
+      - --max-model-len 131072 --enable-expert-parallel --data-parallel-size 8 --enable-auto-tool-choice --tool-call-parser qwen3_coder
+      - Env: VLLM_USE_DEEP_GEMM=1
+      - Notes: Use data-parallel mode (not tensor-parallel) to avoid weight quantization errors. DeepGEMM recommended.
+- [ ] Qwen3-Coder-30B-A3B-Instruct (BF16)
+  - HF: Qwen/Qwen3-Coder-30B-A3B-Instruct
+  - Hardware:
+    - 1x H100/H200
+      - --enable-auto-tool-choice --tool-call-parser qwen3_coder
+      - Notes: Fits comfortably on single GPU. ~60GB model weight.
+    - 2x H100/H200
+      - --tensor-parallel-size 2 --enable-auto-tool-choice --tool-call-parser qwen3_coder
+      - Notes: For higher throughput/longer context.
+- [ ] Qwen3-Coder-30B-A3B-Instruct-FP8
+  - HF: Qwen/Qwen3-Coder-30B-A3B-Instruct-FP8
+  - Hardware:
+    - 1x H100/H200
+      - --enable-auto-tool-choice --tool-call-parser qwen3_coder
+      - Env: VLLM_USE_DEEP_GEMM=1
+      - Notes: FP8 quantized, ~30GB model weight. Excellent for single GPU deployment.
+
+### GPT-OSS
+- Notes: Requires vLLM 0.10.1+gptoss. Built-in tools via /v1/responses endpoint (browsing, Python). Function calling not yet supported. --async-scheduling recommended for higher perf (not compatible with structured output).
+- [ ] GPT-OSS-20B
+  - HF: openai/gpt-oss-20b
+  - Hardware:
+    - 1x H100/H200
+      - --async-scheduling
+    - 1x B200
+      - --async-scheduling
+      - Env: VLLM_USE_TRTLLM_ATTENTION=1 VLLM_USE_TRTLLM_DECODE_ATTENTION=1 VLLM_USE_TRTLLM_CONTEXT_ATTENTION=1 VLLM_USE_FLASHINFER_MXFP4_MOE=1
+- [ ] GPT-OSS-120B
+  - HF: openai/gpt-oss-120b
+  - Hardware:
+    - 1x H100/H200
+      - --async-scheduling
+      - Notes: Needs --gpu-memory-utilization 0.95 --max-num-batched-tokens 1024 to avoid OOM
+    - 2x H100/H200
+      - --tensor-parallel-size 2 --async-scheduling
+      - Notes: Set --gpu-memory-utilization <0.95 to avoid OOM
+    - 4x H100/H200
+      - --tensor-parallel-size 4 --async-scheduling
+    - 8x H100/H200
+      - --tensor-parallel-size 8 --async-scheduling --max-model-len 131072 --max-num-batched-tokens 10240 --max-num-seqs 128 --gpu-memory-utilization 0.85 --no-enable-prefix-caching
+    - 1x B200
+      - --async-scheduling
+      - Env: VLLM_USE_TRTLLM_ATTENTION=1 VLLM_USE_TRTLLM_DECODE_ATTENTION=1 VLLM_USE_TRTLLM_CONTEXT_ATTENTION=1 VLLM_USE_FLASHINFER_MXFP4_MOE=1
+    - 2x B200
+      - --tensor-parallel-size 2 --async-scheduling
+      - Env: VLLM_USE_TRTLLM_ATTENTION=1 VLLM_USE_TRTLLM_DECODE_ATTENTION=1 VLLM_USE_TRTLLM_CONTEXT_ATTENTION=1 VLLM_USE_FLASHINFER_MXFP4_MOE=1
+
+### GLM-4.5
+- Notes: Listed configs support reduced context. For full 128K context, double the GPU count. Models default to thinking mode (disable with API param).
+- [ ] GLM-4.5 (BF16)
+  - HF: zai-org/GLM-4.5
+  - Hardware:
+    - 16x H100
+      - --tensor-parallel-size 16 --tool-call-parser glm45 --reasoning-parser glm45 --enable-auto-tool-choice
+    - 8x H200
+      - --tensor-parallel-size 8 --tool-call-parser glm45 --reasoning-parser glm45 --enable-auto-tool-choice
+  - Notes: On 8x H100, may need --cpu-offload-gb 16 to avoid OOM. For full 128K: needs 32x H100 or 16x H200.
+- [ ] GLM-4.5-FP8
+  - HF: zai-org/GLM-4.5-FP8
+  - Hardware:
+    - 8x H100
+      - --tensor-parallel-size 8 --tool-call-parser glm45 --reasoning-parser glm45 --enable-auto-tool-choice
+    - 4x H200
+      - --tensor-parallel-size 4 --tool-call-parser glm45 --reasoning-parser glm45 --enable-auto-tool-choice
+  - Notes: For full 128K context: needs 16x H100 or 8x H200.
+- [ ] GLM-4.5-Air (BF16)
+  - HF: zai-org/GLM-4.5-Air
+  - Hardware:
+    - 4x H100
+      - --tensor-parallel-size 4 --tool-call-parser glm45 --reasoning-parser glm45 --enable-auto-tool-choice
+    - 2x H200
+      - --tensor-parallel-size 2 --tool-call-parser glm45 --reasoning-parser glm45 --enable-auto-tool-choice
+  - Notes: For full 128K context: needs 8x H100 or 4x H200.
+- [ ] GLM-4.5-Air-FP8
+  - HF: zai-org/GLM-4.5-Air-FP8
+  - Hardware:
+    - 2x H100
+      - --tensor-parallel-size 2 --tool-call-parser glm45 --reasoning-parser glm45 --enable-auto-tool-choice
+    - 1x H200
+      - --tensor-parallel-size 1 --tool-call-parser glm45 --reasoning-parser glm45 --enable-auto-tool-choice
+  - Notes: For full 128K context: needs 4x H100 or 2x H200.
+
+### Kimi
+- Notes: Requires vLLM v0.10.0rc1+. Minimum 16 GPUs for FP8 with 128k context. Reuses DeepSeekV3 architecture with model_type="kimi_k2".
+- [ ] Kimi-K2-Instruct
+  - HF: moonshotai/Kimi-K2-Instruct
+  - Hardware:
+    - 16x H200/H20
+      - --tensor-parallel-size 16 --trust-remote-code --enable-auto-tool-choice --tool-call-parser kimi_k2
+      - Notes: Pure TP mode. For >16 GPUs, combine with pipeline-parallelism.
+    - 16x H200/H20 (DP+EP mode)
+      - --data-parallel-size 16 --data-parallel-size-local 8 --enable-expert-parallel --max-num-batched-tokens 8192 --max-num-seqs 256 --gpu-memory-utilization 0.85 --trust-remote-code --enable-auto-tool-choice --tool-call-parser kimi_k2
+      - Notes: Data parallel + expert parallel mode for higher throughput. Requires multi-node setup with proper networking.
+

package/docs/pi.dev/pods-docs/plan.md

@@ -0,0 +1,166 @@
+## Pi
+
+Pi automates vLLM deployment on GPU pods from DataCrunch, Vast.ai, Prime Intellect, RunPod (or any Ubuntu machine with NVIDIA GPUs). It manages multiple concurrent model deployments via separate vLLM instances, each accessible through the OpenAI API protocol with API key authentication.
+
+Pods are treated as ephemeral - spin up when needed, tear down when done. To avoid re-downloading models (30+ minutes for 100GB+ models), pi uses persistent network volumes for model storage that can be shared across pods on the same provider. This minimizes both cost (only pay for active compute) and setup time (models already cached).
+
+## Usage
+
+### Pods
+```bash
+pi pods setup dc1 "ssh root@1.2.3.4" --mount "mount -t nfs..."  # Setup pod (requires HF_TOKEN, PI_API_KEY env vars)
+pi pods                              # List all pods (* = active)
+pi pods active dc2                   # Switch active pod
+pi pods remove dc1                   # Remove pod
+```
+
+### Models
+```bash
+pi start Qwen/Qwen2.5-72B-Instruct --name qwen72b          # Known model - pi handles vLLM args
+pi start some/unknown-model --name mymodel --vllm --tensor-parallel-size 4 --max-model-len 32768  # Custom vLLM args
+pi list                              # List running models with ports
+pi stop qwen72b                      # Stop model
+pi logs qwen72b                      # View model logs
+```
+
+For known models, pi automatically configures appropriate vLLM arguments from model documentation based on the hardware of the pod. For unknown models or custom configurations, pass vLLM args after `--vllm`.
+
+## Pod management
+
+Pi manages GPU pods from various providers (DataCrunch, Vast.ai, Prime Intellect, RunPod) as ephemeral compute resources. Users manually create pods via provider dashboards, then register them with pi for automated setup and management.
+
+Key capabilities:
+- **Pod setup**: Transform bare Ubuntu/Debian machines into vLLM-ready environments in ~2 minutes
+- **Model caching**: Optional persistent storage shared by pods to avoid re-downloading 100GB+ models
+- **Multi-pod management**: Register multiple pods, switch between them, maintain different environments
+
+### Pod setup
+
+When a user creates a fresh pod on a provider, they register it with pi using the SSH command from the provider:
+
+```bash
+pi pods setup dc1 "ssh root@1.2.3.4" --mount "mount -t nfs..."
+```
+
+This copies and executes `pod_setup.sh` which:
+1. Detects GPUs via `nvidia-smi` and stores count/memory in local config
+2. Installs CUDA toolkit matching the driver version
+3. Creates Python environment
+   - Installs uv and Python 3.12
+   - Creates venv at ~/venv with PyTorch (--torch-backend=auto)
+   - Installs vLLM (model-specific versions when needed)
+   - Installs FlashInfer (builds from source if required)
+   - Installs huggingface-hub (for model downloads)
+   - Installs hf-transfer (for accelerated downloads)
+4. Mounts persistent storage if provided
+   - Symlinks to ~/.cache/huggingface for model caching
+5. Configures environment variables persistently
+
+Required environment variables:
+- `HF_TOKEN`: HuggingFace token for model downloads
+- `PI_API_KEY`: API key for securing vLLM endpoints
+
+### Model caching
+
+Models can be 100GB+ and take 30+ minutes to download. The `--mount` flag enables persistent model caching:
+
+- **DataCrunch**: NFS shared filesystems, mountable across multiple running pods in same region
+- **RunPod**: Network volumes persist independently but cannot be shared between running pods
+- **Vast.ai**: Volumes locked to specific machine - no sharing
+- **Prime Intellect**: No persistent storage documented
+
+Without `--mount`, models download to pod-local storage and are lost on termination.
+
+### Multi-pod management
+
+Users can register multiple pods and switch between them:
+
+```bash
+pi pods                    # List all pods (* = active)
+pi pods active dc2         # Switch active pod
+pi pods remove dc1         # Remove pod from local config but doesn't destroy pod remotely.
+```
+
+All model commands (`pi start`, `pi stop`, etc.) target the active pod, unless `--pod <podname>` is given, which overrides the active pod for that command.
+
+## Model deployment
+
+Pi uses direct SSH commands to manage vLLM instances on pods. No remote manager component is needed - everything is controlled from the local pi CLI.
+
+### Architecture
+The pi CLI maintains all state locally in `~/.pi/pods.json`:
+```json
+{
+  "pods": {
+    "dc1": {
+      "ssh": "ssh root@1.2.3.4",
+      "gpus": [
+        {"id": 0, "name": "H100", "memory": "80GB"},
+        {"id": 1, "name": "H100", "memory": "80GB"}
+      ],
+      "models": {
+        "qwen": {
+          "model": "Qwen/Qwen2.5-72B",
+          "port": 8001,
+          "gpu": "0",
+          "pid": 12345
+        }
+      }
+    }
+  },
+  "active": "dc1"
+}
+```
+
+The location of the pi config dir can also be specified via the `PI_CONFIG_DIR` env var, e.g. for testing.
+
+Pods are assumed to be fully managed by pi - no other processes compete for ports or GPUs.
+
+### Starting models
+When user runs `pi start Qwen/Qwen2.5-72B --name qwen`:
+1. CLI determines next available port (starting from 8001)
+2. Selects GPU (round-robin based on stored GPU info)
+3. Downloads model if not cached:
+   - Sets `HF_HUB_ENABLE_HF_TRANSFER=1` for fast downloads
+   - Runs via SSH with output piped to local terminal
+   - Ctrl+C cancels download and returns control
+4. Builds vLLM command with appropriate args and PI_API_KEY
+5. Executes via SSH: `ssh pod "nohup vllm serve ... > ~/.vllm_logs/qwen.log 2>&1 & echo $!"`
+6. Waits for vLLM to be ready (checks health endpoint)
+7. On success: stores port, GPU, PID in local state
+8. On failure: shows exact error from vLLM logs, doesn't save to config
+
+### Managing models
+- **List**: Show models from local state, optionally verify PIDs still running
+- **Stop**: SSH to kill process by PID
+- **Logs**: SSH to tail -f log files (Ctrl+C stops tailing, doesn't kill vLLM)
+
+### Error handling
+- **SSH failures**: Prompt user to check connection or remove pod from config
+- **Stale state**: Commands that fail with "process not found" auto-clean local state
+- **Setup failures**: Ctrl+C during setup kills remote script and exits cleanly
+
+### Testing models
+The `pi prompt` command provides a quick way to test deployed models:
+```bash
+pi prompt qwen "What is 2+2?"                    # Simple prompt
+pi prompt qwen "Read file.txt and summarize"     # Uses built-in tools
+```
+
+Built-in tools for agentic testing:
+- `ls(path, ignore?)`: List files and directories at path, with optional ignore patterns
+- `read(file_path, offset?, limit?)`: Read file contents with optional line offset/limit
+- `glob(pattern, path?)`: Find files matching glob pattern (e.g., "**/*.py", "src/**/*.ts")
+- `rg(args)`: Run ripgrep with any arguments (e.g., "pattern -t py -C 3", "TODO --type-not test")
+
+The provided prompt will be augmented with info on the current local working directory. File tools expect absolute paths.
+
+This allows testing basic agent capabilities without external tool configuration.
+
+`prompt` is implemented using the latest OpenAI SDK for NodeJS. It outputs thinking content, tool calls and results, and normal assistant messages.
+
+## Models
+We want to support these models specifically, with alternative models being marked as "possibly works". This list will be updated with new models regularly. A checked
+box means "supported".
+
+See [models.md](./models.md) for a list of models, their HW reqs, vLLM args and notes, we want to support out of the box with a simple `pi start <model-name> --name <local-name>`

package/docs/pi.dev/pods-docs/qwen3-coder.md

@@ -0,0 +1,132 @@
+# Qwen3-Coder Usage Guide
+
+[Qwen3-Coder](https://github.com/QwenLM/Qwen3-Coder) is an advanced large language model created by the Qwen team from Alibaba Cloud. vLLM already supports Qwen3-Coder, and `tool-call` functionality will be available in vLLM v0.10.0 and higher You can install vLLM with `tool-call` support using the following method:
+
+## Installing vLLM
+
+```bash
+uv venv
+source .venv/bin/activate
+uv pip install -U vllm --torch-backend auto
+```
+
+## Launching Qwen3-Coder with vLLM
+
+### Serving on 8xH200 (or H20) GPUs (141GB × 8)
+
+**BF16 Model**
+
+```bash
+vllm serve Qwen/Qwen3-Coder-480B-A35B-Instruct \
+  --tensor-parallel-size 8 \
+  --max-model-len 32000 \
+  --enable-auto-tool-choice \
+  --tool-call-parser qwen3_coder
+```
+
+**FP8 Model**
+
+```bash
+VLLM_USE_DEEP_GEMM=1 vllm serve Qwen/Qwen3-Coder-480B-A35B-Instruct-FP8 \
+  --max-model-len 131072 \
+  --enable-expert-parallel \
+  --data-parallel-size 8 \
+  --enable-auto-tool-choice \
+  --tool-call-parser qwen3_coder
+```
+
+## Performance Metrics
+
+### Evaluation
+We launched `Qwen3-Coder-480B-A35B-Instruct-FP8` using vLLM and evaluated its performance using  [EvalPlus](https://github.com/evalplus/evalplus). The results are displayed below:
+
+| Dataset | Test Type | Pass@1 Score |
+|-----------|-----------|--------------|
+| HumanEval | Base tests | 0.939 |
+| HumanEval+ | Base + extra tests | 0.902 |
+| MBPP | Base tests | 0.918 |
+| MBPP+ | Base + extra tests | 0.794 |
+
+### Benchmarking
+We used the following script to benchmark `Qwen/Qwen3-Coder-480B-A35B-Instruct-FP8`
+
+```bash
+vllm bench serve \
+  --backend vllm \
+  --model Qwen/Qwen3-Coder-480B-A35B-Instruct-FP8 \
+  --endpoint /v1/completions \
+  --dataset-name random \
+  --random-input 2048 \
+  --random-output 1024 \
+  --max-concurrency 10 \
+  --num-prompt 100 \
+```
+If successful, you will see the following output.
+
+```shell
+============ Serving Benchmark Result ============
+Successful requests:                     100
+Benchmark duration (s):                  776.49
+Total input tokens:                      204169
+Total generated tokens:                  102400
+Request throughput (req/s):              0.13
+Output token throughput (tok/s):         131.88
+Total Token throughput (tok/s):          394.81
+---------------Time to First Token----------------
+Mean TTFT (ms):                          7639.31
+Median TTFT (ms):                        6935.71
+P99 TTFT (ms):                           13766.68
+-----Time per Output Token (excl. 1st token)------
+Mean TPOT (ms):                          68.43
+Median TPOT (ms):                        67.23
+P99 TPOT (ms):                           72.14
+---------------Inter-token Latency----------------
+Mean ITL (ms):                           68.43
+Median ITL (ms):                         66.34
+P99 ITL (ms):                            69.38
+==================================================
+
+```
+
+
+## Using Tips
+
+### BF16 Models
+- **Context Length Limitation**: A single H20 node cannot serve the original context length (262144). You can reduce the `max-model-len` or increase `gpu-memory-utilization` to work within memory constraints.
+
+### FP8 Models
+- **Context Length Limitation**: A single H20 node cannot serve the original context length (262144). You can reduce the `max-model-len` or increase `gpu-memory-utilization` to work within memory constraints.
+- **DeepGEMM Usage**: To use [DeepGEMM](https://github.com/deepseek-ai/DeepGEMM), set `VLLM_USE_DEEP_GEMM=1`. Follow the [setup instructions](https://github.com/vllm-project/vllm/blob/main/benchmarks/kernels/deepgemm/README.md#setup) to install it.
+- **Tensor Parallelism Issue**: When using `tensor-parallel-size 8`, the following failures are expected. Switch to data-parallel mode using `--data-parallel-size`.
+- **Additional Resources**: Refer to the [Data Parallel Deployment documentation](https://docs.vllm.ai/en/latest/serving/data_parallel_deployment.html) for more parallelism groups.
+
+```shell
+ERROR [multiproc_executor.py:511]   File "/vllm/vllm/model_executor/models/qwen3_moe.py", line 336, in <lambda>
+ERROR [multiproc_executor.py:511]     lambda prefix: Qwen3MoeDecoderLayer(config=config,
+ERROR [multiproc_executor.py:511]                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+ERROR [multiproc_executor.py:511]   File "/vllm/vllm/model_executor/models/qwen3_moe.py", line 278, in __init__
+ERROR [multiproc_executor.py:511]     self.mlp = Qwen3MoeSparseMoeBlock(config=config,
+ERROR [multiproc_executor.py:511]                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+ERROR [multiproc_executor.py:511]   File "/vllm/vllm/model_executor/models/qwen3_moe.py", line 113, in __init__
+ERROR [multiproc_executor.py:511]     self.experts = FusedMoE(num_experts=config.num_experts,
+ERROR [multiproc_executor.py:511]                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+ERROR [multiproc_executor.py:511]   File "/vllm/vllm/model_executor/layers/fused_moe/layer.py", line 773, in __init__
+ERROR [multiproc_executor.py:511]     self.quant_method.create_weights(layer=self, **moe_quant_params)
+ERROR [multiproc_executor.py:511]   File "/vllm/vllm/model_executor/layers/quantization/fp8.py", line 573, in create_weights
+ERROR [multiproc_executor.py:511]     raise ValueError(
+ERROR [multiproc_executor.py:511] ValueError: The output_size of gate's and up's weight = 320 is not divisible by weight quantization block_n = 128.
+```
+
+### Tool Calling
+- **Enable Tool Calls**: Add `--tool-call-parser qwen3_coder` to enable tool call parsing functionality, please refer to: [tool_calling](https://docs.vllm.ai/en/latest/features/tool_calling.html)
+
+## Roadmap
+
+- [x] Add benchmark results
+
+
+## Additional Resources
+
+- [EvalPlus](https://github.com/evalplus/evalplus)
+- [Qwen3-Coder](https://github.com/QwenLM/Qwen3-Coder)
+- [vLLM Documentation](https://docs.vllm.ai/)